January 2019
This document sets out the assessment criteria and working methods of the main and sub-
panels for the Research Excellence Framework (REF) 2021.
Panel criteria and
working methods
Department for the Economy, Northern Ireland
Higher Education Funding Council for Wales
Research England
Scottish Funding Council
1
Contents
Contents ....................................................................................................................................... 1
Executive summary ...................................................................................................................... 2
Introduction ................................................................................................................................... 4
Part 1: Overview ........................................................................................................................... 5
Part 2: Unit of assessment descriptors ...................................................................................... 12
Part 3: Assessment criteria ........................................................................................................ 37
Part 4: Panel procedures............................................................................................................ 83
Part 5: Panel working methods .................................................................................................. 84
Annex A: Examples of impacts and indicators ........................................................................... 93
Annex B: Summary of additional information about outputs .................................................... 110
Annex C: Main Panel D output types & submission guidance.............................................. 113
Annex D: Managing conflicts of interest ................................................................................... 120
Annex E: Confidentiality and information security agreement for REF 2021 panels............... 124
Annex F: List of abbreviations .................................................................................................. 127
2
REF 2021: Panel criteria and working methods
To
Heads of higher education institutions in the UK
Of interest to those
responsible for
Research
Reference
REF 2019/02
Publication date
January 2019
Enquiries from staff at UK
higher education
institutions
Email your institutional REF contact. (These are listed at www.ref.ac.uk,
under Contact.)
Other enquiries
Hannah Daisley, tel 0117 931 7486, email info@ref.ac.uk
Executive summary
Purpose
1. This document sets out the assessment criteria and working methods of the main and sub-
panels for the Research Excellence Framework (REF) 2021.
2. This document, taken together with the ‘Guidance on submissions’ (REF 2019/01), provides a
comprehensive description of the information required in submissions to the REF, and how the REF
panels will assess submissions.
Key points
3. The REF is a process of expert review. Expert sub-panels for each of 34 units of assessment
(UOA) will carry out the assessment, working under the leadership and guidance of four main
panels.
4. UK higher education institutions (HEIs) will be invited to make submissions by 27 November
2020. The REF main and sub-panels will assess submissions during 2021, and results will be
published in December 2021. The results will inform the allocation of research funding by the UK
higher education (HE) funding bodies, from 202223.
5. This document sets out a combined statement of criteria and working methods across the four
main panels. Supplementary criteria are set out for each main panel where applicable. The main
panels’ supplementary criteria are intended to be read alongside the combined criteria, and do not
replace it.
Action required
6. This document is for information and to guide institutions in preparing and collecting data for
inclusion in REF submissions. No action is required by HEIs at this stage.
3
Further information
7. For further information about the REF see www.ref.ac.uk.
8. Staff at UK HEIs should direct any queries to their institutional REF contact. Contact details for
each institution are listed at www.ref.ac.uk, under Contact.
9. Other enquiries should be addressed to info@ref.ac.uk.
4
Introduction
10. This document sets out the assessment criteria and working methods of the main and sub-
panels for REF 2021.
11. The criteria have been developed as a combined statement across the four main panels. This
responds to feedback following REF 2014 that there was further scope for increased consistency in
the criteria, and incorporates advice from the main and sub-panels on where further alignment
across the criteria could be achieved. Supplementary criteria are set out for each main panel where
applicable. The main panels’ supplementary criteria are intended to be read alongside the combined
criteria, and do not replace it.
12. In early 2018, the REF team invited the four main panels to develop their criteria and working
methods, with input from their sub-panels. The REF team provided guidance to the panels on
developing their criteria. The ‘Guidance to panels’ is available at www.ref.ac.uk, under Publications.
This invited the main panels to develop a combined set of criteria and working methods, with
supplementary criteria for each main panel and its group of sub-panels where applicable. The main
panels were invited to agree differences in approaches for particular sub-panels within their remit
only where justified by differences in the nature of research in those disciplines.
13. The assessment criteria and working methods are set out in this document as follows:
Part 1 provides an overview of the REF and the expert panels.
Part 2 sets out the descriptor for each of the 34 units of assessment (UOAs).
Part 3 gives further details of the assessment criteria to be employed by the four main panels
and their sub-panels, including the combined criteria and, where applicable, any
supplementary criteria provided by any main panel.
Part 4 sets out the generic panel procedures in place across the main and sub-panels.
Part 5 sets out the combined working methods of the main and sub-panels.
14. This document should be read alongside REF 2019/01 Guidance on submissions’ (hereafter
‘Guidance on submissions’), which provides an overview of the REF assessment framework and
guidance to institutions on preparing their submissions, including the data requirements and
definitions that apply.
15. Together, the two documents will give a comprehensive description of the information required
in submissions to the REF, and how the REF panels will assess submissions. We will issue
supplements to the guidance at later dates to clarify points of detail regarding submissions. Panels
will not be permitted to depart from the final criteria once published, other than in exceptional
circumstances that cannot be accommodated within the published framework. In such cases, we will
publish the reason and details of the change as an amendment.
5
Part 1: Overview
16. The REF is the system for assessing the quality of research in HEIs in the UK. It was first
conducted in 2014, and replaced the previous Research Assessment Exercise (RAE).
17. The purpose of the REF, the general principles governing its conduct, and an overview of the
REF framework are set out in Part 1 of ‘Guidance on submissions’.
Submissions and units of assessment
18. Institutions will be invited to make submissions by 27 November 2020. There are 34 UOAs,
listed in Annex D of ‘Guidance on submissions’. Part 2 of this document provides descriptors of each
UOA. Each submission must contain, in summary:
a. REF1a/b: Information on all staff in post on the census date, 31 July 2020, with
significant responsibility for research; and information about former staff to whom
submitted outputs are attributed.
b. REF2: Details of assessable outputs that the submitted unit has produced during the
publication period (1 January 2014 to 31 December 2020). The total number of outputs
must equal 2.5 times the summed full-time equivalent (FTE) of the unit’s submitted staff.
Rounding to the nearest whole number will be applied to give a whole number of outputs
for submission.
c. REF3: Case studies describing specific examples of impacts achieved during the
assessment period (1 August 2013 to 31 July 2020), underpinned by excellent research
in the period 1 January 2000 to 31 December 2020.
d. REF4a/b/c: Data about research doctoral degrees awarded, research income and
income-in-kind related to the period 1 August 2013 to 31 July 2020.
e. REF5a/b: An institutional-level environment statement, and a completed template
describing the submitted unit’s research and impact environment, drawing on
quantitative indicators as appropriate, and related to the period 1 August 2013 to 31 July
2020.
f. REF6a/b: Information on staff circumstances, where claiming a reduction or removal of
the requirement to submit a minimum of one output per member of staff submitted.
19. The generic eligibility definitions and data requirements set out in ‘Guidance on submissions’
apply to all submissions.
6
Expert panels
20. The REF will be a process of expert review, with an expert panel assessing submissions made
by HEIs in each of the 34 UOAs. The sub-panels will work under the leadership and guidance of four
main panels.
21. In brief, the four main panels are responsible for:
developing the panel criteria and working methods
overseeing calibration
ensuring adherence to the published procedures and consistent application of the overall
assessment standards by the sub-panels
working with the advisory panels for equality and diversity and interdisciplinary research
(IDR), and advising the REF team on assessment processes
signing off the outcomes of the assessment.
22. The sub-panels are responsible for:
contributing to the criteria and working methods
working within the agreed criteria in assessing each submission made in its UOA
recommending the outcomes for each submission to the main panel.
23. The roles and responsibilities of the main and sub-panels are described fully in ‘Roles and
recruitment of the expert panels’ (REF 2017/03).
24. The main and sub-panels will undertake their roles within the common framework for
assessment set out in ‘Guidance on submissions’ and the combined statement of criteria and
working methods (Parts 2 to 5 of this document). Part 3 of this document also sets out any
supplementary criteria that each of the main panels and its sub-panels will employ when assessing
submissions.
25. The main and sub-panels are appointed by the four UK funding bodies through an open
process of nominations, as described in REF 2017/03. The appointment of the expert panels will be
undertaken in two stages for REF 2021:
a. Criteria phase: sufficient members were appointed in 2018 to ensure each sub-panel
has appropriate expertise (including in IR and the wider use of research) for contributing
to the criteria development. This is a smaller group of members than the volume
required to undertake the assessment, meaning that further full sub-panel members
(and impact and output assessors) will join the sub-panels for the assessment phase.
7
The members joining at the criteria phase will input into the panel criteria development,
and will contribute to the assessment of all elements of submissions (outputs, impact
and environment) in 20202021.
b. Assessment phase: additional sub-panel members will be appointed for the
assessment phase of the exercise, to ensure the sub-panel has an appropriate volume
of members who will contribute to the assessment of all elements of submissions.
26. The membership of the panels appointed for the criteria phase is at www.ref.ac.uk, under
Expert Panels. Following advice from main and sub-panel chairs, where suitable nominations were
received the funding bodies have appointed a proportion of the members who will join the sub-
panels for the assessment phase. These members will take up their panel roles in 2020, and are
also detailed on the REF website.
Appointment of additional sub-panel members and assessors
27. We will make further appointments to the panels of both sub-panel members and assessors in
advance of the assessment year. This is to ensure the sub-panels have an appropriate breadth of
expertise and number of panel members necessary for the assessment phase. We also expect to
make further appointments of members with the role of interdisciplinary adviser in the assessment
phase. This role is described in REF 2017/03 (Box 1).
28. A survey of institutions’ submission intentions will be carried out in 2019, which will inform
additional panel membership appointments, including for both further full sub-panel members as well
as output and impact assessors.
29. Further members and assessors will be appointed by the chief executives (or equivalent) of
the four UK funding bodies, following recommendations from main and sub-panel chairs, made from
nominated individuals. These will either be individuals with appropriate expertise who were
nominated in the initial round of nominations in 2017 (see REF 2017/03), or additional nominations
that the REF team will invite in a further round in 2020. This will include nominations for those with
expertise in the use or benefits of research across the private, public and third sectors.
30. Where further nominations are invited, the REF team will again ask nominating bodies to
provide information on how equality and diversity considerations were taken account of during the
nominations process. The Equality and Diversity Advisory Panel (EDAP) have reviewed the
templates provided by nominating bodies in the 2017 round of nominations, and will produce a report
summarising good practice identified in the nominations process. Nominating bodies will be invited
to consider this report when making nominations in 2020. In recommending further panel members
and assessors, sub-panel chairs will give consideration to enhancing the extent to which the overall
body of members reflects the diversity of the research community. The REF team are currently
undertaking an analysis of the representativeness of panel membership, which will be able to further
inform chairs’ considerations in this regard.
8
Assessors
31. Assessors appointed to the sub-panels will undertake either one of the following roles:
a. To assess the impact element of submissions and develop the impact sub-profiles,
alongside sub-panel members. These will be people with professional experience of
making use of, applying or benefiting from academic research.
b. To assess research outputs and develop the output sub-profiles, alongside existing
panel members. These will be practising researchers with relevant expertise.
32. Assessors will play a full and equal role to sub-panel members in developing the sub-profiles
for either the impact or outputs element of the assessment. They will be fully briefed, take part in
calibration exercises and attend panel meetings at which the relevant aspects of submissions
(outputs or impact) are discussed.
Appointment process
33. Main and sub-panel chairs’ recommendations for further membership will be guided by the
principle of ensuring that sub-panels have access to appropriate expertise to reach robust and valid
judgements with regard to submitted material. Appointments will be made as follows:
a. In 2019, the REF team will survey institutions about the volume and nature of work that
they intend to submit to the REF. The survey data will provide information to the REF
team and the panels about how the changes to the submission process in REF 2021
may affect the volume and range of work submitted, compared with previous exercises.
b. In early 2020, the main and sub-panels will consider the breadth and depth of expertise
of each sub-panel’s current membership, in the light of institutions’ submission
intentions. Each sub-panel will seek to identify:
where the membership required to assess submissions should be expanded to
ensure appropriate expertise in accordance with the anticipated volume of
submissions
where additional IDR advisers may be required to support and advise the sub-panel
on the assessment of IDR outputs
areas where additional expertise would be required to assess the range and volume
of outputs indicated in the survey responses
areas where additional user expertise would be required to assess the range of
impacts indicated in the survey responses.
9
c. Where a clear gap in the expertise of a sub-panel required to assess either outputs or
impact has been identified during the criteria development and consultation phase,
additional appointments may be made during 2019.
34. Before recommending the appointment of further members and assessors, sub-panel chairs
will discuss the recommendations with their main panels. The following issues will be considered
across each main panel:
the identified expertise requirements in the context of the current panel membership’s
breadth and volume of expertise
the overall size of the sub-panel in the context of anticipated submissions
the potential for interdisciplinary advisers to work across sub-panels
the need to ensure that impact case studies are given fair consideration, with the intention of
ensuring that there is sufficient user expertise to review the range of likely impact case
studies that will be submitted
where assessors are recommended:
o whether there is a sufficient body of activity requiring an additional assessor appointment
o whether serious workload issues or conflicts of interest for existing panel members have
been identified, requiring an additional assessor for a particular subject area
o the potential for individual assessors to be appointed to two sub-panels, where there is a
significant overlapping body of work expected (and, if appropriate, the potential to
appoint existing user members to also act as assessors for other sub-panels).
35. Once appointed at each stage, the names of sub-panel members and assessors will be
published on www.ref.ac.uk alongside the current panel membership. Assessors will be eligible to
receive fees and expenses on the same basis as panel members.
36. As stated in REF 2017/03 (paragraph 67), as the REF progresses, main or sub-panels may
recommend to the funding bodies the appointment of a small number of members or assessors in
addition to the members already appointed and/or the members and assessors to be appointed
through the processes outlined above, to provide further expertise where this is desirable and in
accordance with the criteria for appointments.
Assessing submissions
37. As with the previous REF and RAEs, the assessment process is based on expert review. Each
sub-panel will examine the submissions made in its UOA, taking into account all the evidence
10
presented. Each sub-panel will use its professional collective judgement to form an overall view
about each submission and recommend to the main panel an overall quality profile to be awarded to
each submission made in its UOA. The process is objective and evidence based but is not and
cannot be purely algorithmic.
38. The primary outcome of the panels’ work will be an overall quality profile awarded to each
submission, to be published in December 2021. An example overall quality profile is in Annex B of
‘Guidance on submissions’, and further details about the published outcomes and feedback that
panels will produce are in paragraphs 34 to 43 of that document.
39. In forming their overall quality judgements, the sub-panels will assess three distinct elements
of each submission outputs, impact and environment against the following generic criteria:
a. Outputs: The sub-panels will assess the quality of submitted research outputs in terms
of their ‘originality, significance and rigour’, with reference to international research
quality standards. This element will carry a weighting of 60 per cent in the overall
outcome awarded to each submission.
b. Impact: The sub-panels will assess the ‘reach and significance’ of impacts on the
economy, society, culture, public policy or services, health, the environment or quality of
life that were underpinned by excellent research conducted in the submitted unit. This
element will carry a weighting of 25 per cent.
c. Environment: The sub-panels will assess the research environment in terms of its
‘vitality and sustainability’, including the approach to enabling impact from its research,
and its contribution to the vitality and sustainability of the wider discipline or research
base. This element will carry a weighting of 15 per cent.
40. The generic definitions of the starred quality levels in the overall quality profile in each of the
three sub-profiles outputs, impact and environment are in Annex A of ‘Guidance on
submissions’. All sub-panels will apply these generic assessment criteria, level definitions and
weightings for each element, in forming the overall quality profiles to recommend to their main panel.
41. In Part 3 of this document, the main panels’ criteria statement provides a descriptive account
of these generic assessment criteria, and of the starred level definitions for outputs. These are
provided to inform their subject communities on how the panels will apply the criteria and definitions
in making their judgements. These descriptive accounts should be read alongside, but do not
replace, the generic definitions.
Outputs
42. As set out in ‘Guidance on submissions’, each submission must include a set number of items
of research output, equal to 2.5 times the combined FTE of Category A submitted staff included in
the submission. Rounding to the nearest whole number will be applied to give a whole number of
11
outputs for submission. The total number of outputs for return may be adjusted, as appropriate, to
take account of staff circumstances (see ‘Guidance on submissions’, paragraphs 151 to 201).
43. An underpinning principle of the REF is that for each discipline all types of research and all
forms of research output shall be assessed on a fair and equal basis. Panels have been instructed to
define criteria and adopt assessment processes that enable them to recognise, and treat on an
equal footing, excellence in research across the spectrum of applied, practice, basic and strategic
research, wherever that research is conducted; and for identifying excellence in different forms of
research endeavour including interdisciplinary and collaborative research, while attaching no greater
weight to one form over another.
44. ‘Guidance on submissions’ (Annex C) sets out the generic definition of research and its Annex
K provides a glossary of the different output types that may be submitted, and is intended to highlight
the diversity of research outputs that may be submitted in any UOA. Any assessable form of output
that embodies research is eligible for assessment.
Impact
45. ‘Guidance on submissions’ sets out the generic submission requirements in relation to impact,
including the number of case studies required in each submission (paragraph 309), the eligibility
criteria for impact case studies (paragraphs 311 to 313), and a template and guidance on completing
impact case studies (Annex G).
46. The generic definition of impact for the REF given in ‘Guidance on submissions’ (Annex C) is
broad, and any impact that meets this definition is eligible for assessment, in any UOA. The panel
criteria statement in Part 3 and examples in Annex A in this document provide some further
descriptions of the diversity of impacts that may apply in their UOAs. These are provided to inform
their subject communities: they should be read alongside, but do not replace, the generic definition
in ‘Guidance on submissions’.
47. The main panels’ criteria statement in Part 3, and the examples in Annex A, provide guidance
on the forms of evidence that would be appropriate for submissions to include in impact case studies
(REF3). The statement in Part 3 also states how the panels will assure that the quality of research
that underpins impact case studies is equivalent to at least two-star quality.
Environment
48. ‘Guidance on submissions’ sets out the generic requirements for the environment element of
submissions, which comprise:
a. standard data on research doctoral degrees awarded, research income and research
income-in-kind (REF4a/b/c)
b. a completed institutional-level environment statement (REF5a)
c. a completed environment template (REF5b).
12
49. In Part 3, the panel criteria provide guidance on the forms of evidence that would be
appropriate for submissions to include in the environment template (REF5b), including any
quantitative indicators that should be provided within REF5b, where applicable. The template for
REF5b is set out in Annex I of the ‘Guidance on submissions’.
50. REF panels will form an environment sub-profile by assessing the information submitted in
REF5b, informed by the data submitted in REF4a/b/c and REF5a. When the REF team provides
submissions to sub-panels, we will supply a standard analysis of the quantitative data submitted in
REF4a/b/c, in respect of each submission in that UOA, and aggregated for all submissions in that
UOA (see ‘Guidance on submissions’, Annex J). Panels will consider these data within the context of
the information provided in REF5b, and within the context of the disciplines concerned. In Part 3, the
panels’ criteria statement indicates how the data analyses will be used in informing the assessment
of the research environment.
Part 2: Unit of assessment descriptors
51. Details of the assessment of interdisciplinary work and work that crosses UOA boundaries,
including pedagogic research, are provided in paragraphs 166 to 177.
Main Panel A: UOAs 16
Introduction
52. The UOAs within Main Panel A’s remit cover research into the practices, services, policies,
education and underpinning science relevant to these disciplines, and associated methodological
and theoretical advancement. The UOAs cover a full spectrum of research approaches, ranging from
qualitative to quantitative, as well as theoretical and mixed method studies. This includes IDR and
research that informs these areas from a range of stakeholders’ perspectives, including research
users and service users.
53. Where it is relevant to the UOAs, submissions will be welcome whether the context is local,
national or international, including work carried out in relation to developing countries.
54. The main panel encourages institutions to structure their submissions using research groups,
noting that there is no expectation that submissions will necessarily comprise a single coherent body
of research. Where submissions are structured using research groups, the sub-panels’ written
qualitative feedback to institutions may highlight individual research groups of particular note.
Unit of assessment descriptors and boundaries
Unit of Assessment 1: Clinical Medicine
55. The UOA includes research into all aspects of clinical medicine and its cognate sub-disciplines
except for bodies of research more explicitly linked to UOA 2 (Public Health, Health Services and
13
Primary Care), UOA 3 (Allied Health Professions, Dentistry, Nursing and Pharmacy), UOA 4
(Psychology, Psychiatry and Neuroscience) and UOA 5 (Biological Sciences).
56. The sub-panel expects submissions that demonstrate integrated strategies relating to all
aspects of medical research. Submissions may cover the full range of research related to medicine,
from basic underpinning studies through experimental medicine to clinical trials. In view of the
breadth of research covered by this UOA, the sub-panel expects some degree of overlap with UOA
4 (Psychology, Psychiatry and Neuroscience) in the fields of neurology and with UOA 5 (Biological
Sciences) in the area of basic biological sciences underpinning medical research.
Unit of Assessment 2: Public Health, Health Services and Primary Care
57. The UOA includes research into all aspects of public health, health services and/or primary
care and all their cognate disciplines. The research may be applied, theoretical or methodological
research from relevant health or healthcare disciplines.
58. The sub-panel expects submissions in this UOA from all areas of public health and
epidemiology (from aetiology to intervention), health services and primary care, including clinical
trials, health social sciences, health policy research and healthcare management, and from other
related disciplines having a relevance to the research covered by the UOA. It recognises the breadth
and diverse range of single, multidisciplinary and/or multi-professional research across public health,
health services and primary care.
Unit of Assessment 3: Allied Health Professions, Dentistry, Nursing and
Pharmacy
59. The UOA includes research into all aspects of the disciplines of allied health professions,
dentistry, nursing, midwifery and pharmacy. Its boundaries include research in underpinning science,
laboratory-based work, applied clinical research, healthcare technologies, and research into public
health, social care and health promotion. Research into psychosocial, philosophical and ethical
aspects of healthcare, as well as education, policy and methodology relevant to these disciplines, is
also included. It is anticipated that such work will use qualitative, quantitative and mixed methods, as
well as theoretical approaches.
60. For allied health professions, submitted research is expected to underpin clinical practice and
policy development and implementation, and includes research in biomedical and nutritional
sciences, dietetics, biology of health and disease, vision sciences, optometry, orthoptics, osteopathy,
operating department practitioners, diagnostic imaging, therapeutic radiography, audiology, podiatry,
occupational therapy, physiotherapy, speech and language therapy, clinical linguistics, paramedics,
prosthetics/orthotics, music therapy, drama therapy and arts therapy. For dentistry, it includes
research in basic and applied dental, oral and craniofacial sciences encompassing all the related
clinical disciplines, primary dental care, biomaterials sciences relevant to oral and craniofacial
science, and other such sciences relevant to dentistry. For nursing and midwifery, it includes
specialist, community and public health nursing, and all the contexts within which they operate. For
pharmacy, it includes all aspects of the design, synthesis, formulation, action and use of
14
pharmaceuticals (including biological and nutraceuticals), to include medicinal chemistry,
pharmaceutics, pharmacology, clinical pharmacy, underlying biomedical science, and the practice of
pharmacy.
61. Submissions may cover the full translational range of research, from basic underpinning
studies through to implementation research. Bodies of research more explicitly linked to UOA 1
(Clinical Medicine), UOA 2 (Public Health, Health Services and Primary Care), UOA 4 (Psychology,
Psychiatry and Neuroscience), UOA 5 (Biological Sciences) and UOA 6 (Agriculture, Food and
Veterinary Sciences) should be submitted to those panels and such outputs submitted to UOA 3 will
be cross-referred.
Unit of Assessment 4: Psychology, Psychiatry and Neuroscience
62. The UOA expects submissions in all areas of psychology, psychiatry and neuroscience.
63. This includes all areas of psychological research with humans and animals. It covers
quantitative and qualitative approaches to typical and atypical populations in all settings at the
cultural, societal, group and individual levels, across the full range of areas of psychological practice
and outside of formalised settings, and includes all aspects of psychological experience. For
psychiatry, it covers research on all aspects of the study of mental disorders. This includes their
aetiology, epidemiology, mechanisms and consequences, as well as pharmacological and non-
pharmacological treatments, systems of care and mental health policy. For neuroscience it covers all
aspects of research from the molecular through to whole-system behavioural level, genetics and
varieties of imaging, and both neurodevelopmental and adult work. This includes research on the
understanding and treatment of all types of neurological and neurosurgical conditions, including
those related to neurodegeneration and neurodevelopment.
64. The remit of the sub-panel is broad, covering submissions with the potential to transform
research into practice as well as those reporting theoretical and methodological advances in basic
research. Overlap is expected with UOA 1 (Clinical Medicine), UOA 2 (Public Health, Health
Services and Primary Care), UOA 3 (Allied Health Professions, Dentistry, Nursing and Pharmacy),
UOA 5 (Biological Sciences), UOA 8 (Chemistry), UOA 9 (Physics), UOA 11 (Computer Science and
Informatics), UOA 12 (Engineering), UOA 17 (Business and Management Studies), UOA 20 (Social
Work and Social Policy), UOA 21 (Sociology), UOA 23 (Education), UOA 24 (Sports and Exercise
Sciences, Leisure and Tourism), UOA 26 (Modern Language and Linguistics) and UOA 30
(Philosophy).
Unit of Assessment 5: Biological Sciences
65. The UOA includes research into all aspects of biological and biomedical sciences that
encompasses the full spectrum of the fundamental and applied biology of all organisms, at all levels
of organisation from the molecular to the ecosystem, employing a diversity of approaches including
experimental, theoretical, computational and mathematical. The UOA also covers all aspects of the
biomedical sciences, including biochemistry, physiology, pharmacology and anatomy at the genetic,
15
molecular, cellular, organ system and whole-organism level. It includes work relevant to the nervous
and cardiovascular systems at all levels of enquiry.
66. Submissions may include work which is on the boundaries of other UOAs in Main Panel A,
such as: UOA 1 (Clinical Medicine), UOA 3 (Allied Health Professions, Dentistry, Nursing and
Pharmacy), UOA 4 (Psychology, Psychiatry and Neuroscience) and UOA 6 (Agriculture, Food and
Veterinary Sciences); as well as UOAs in other main panels, such as: UOA 7 (Earth Systems and
Environmental Sciences), UOA 8 (Chemistry), UOA 9 (Physics), UOA 10 (Mathematical Sciences),
UOA 11 (Computer Science and Informatics), UOA 14 (Geography and Environmental Studies) and
UOA 24 (Sport and Exercise Sciences, Leisure and Tourism).
Unit of Assessment 6: Agriculture, Food and Veterinary Sciences
67. The UOA includes research into all aspects of the agricultural, food and veterinary sciences,
including basic through to applied research, and interdisciplinary research with significant content in
any of these areas of science.
68. The sub-panel expects submissions in this UOA from all areas of relevant science.
a. In agricultural science this includes the scientific exploration of all aspects of agriculture,
horticulture, forestry, aquaculture and marine, including the sciences underpinning and
impacted by them, and the emerging technologies that support these industries with
particular reference to the combination of economic, environmental and social
disciplines into defining optimal systems of land use, the development of the bio-
economy and integrated approaches to farming.
b. For veterinary science, submissions may cover the full range of research related to
veterinary medicine and surgery. These include studies ranging from basic underpinning
research through to clinical, applied and social science. Submissions may include
research relevant to normal and abnormal function of animals, their health, behaviour,
welfare, nutrition and productivity, as well as their role in human health, environment
and society, or as models for human diseases.
c. For food, this includes submissions of primary relevance to food science and technology
(including chemistry, physics, microbiology, engineering and processing), human
nutrition, diet and health, food biotechnology, food safety, packaging, sensory science,
food consumer science, and food security, sustainability and environmental aspects.
69. In view of the breadth of research covered by this UOA, the sub-panel expects some degree
of overlap with underpinning sciences research which may be presented to all panels.
16
Main Panel B: UOAs 712
Introduction
70. The six sub-panels that fall within Main Panel B invite submissions in UOAs 7 to 12 as set out
in the following paragraphs.
71. Institutions should make submissions to the most appropriate UOA. Sub-panels will utilise
cross-referral and interdisciplinary research processes where appropriate.
72. Main Panel B recognises that work in materials science takes places in several of its
disciplines and as a result crosses the boundaries of UOAs 7, 8, 9 or 12.
73. HEIs may choose to associate outputs with research groups if they have used these research
groups to structure their environment submission. This is not a mandatory field and neither the
presence nor absence of research groups is assumed.
Unit of assessment descriptors and boundaries
Unit of Assessment 7: Earth Systems and Environmental Sciences
74. The UOA includes Earth, environmental and planetary sciences, including: geophysics;
geochemistry; palaeontology; geology; mineral physics; evolution of planetary atmospheres,
surfaces and interiors; Earth surface processes; the physics, chemistry and biology of the
environment, including ecology and conservation; atmospheric, marine, freshwater, terrestrial and
soil sciences; innovative measurement systems and data analysis; global change; natural resources;
natural hazards; pollution; and environmental management and impact.
75. The sub-panel expects submissions in this UOA from all areas of Earth systems and
environmental sciences, as defined above, and expects that the majority of the research activity
submitted will have made a direct contribution to the UOA as characterised in the UOA descriptor. It
recognises, however, the increasingly interdisciplinary nature of research, and expects that
submissions may contain work that contributes to this UOA and other cognate disciplines. It is
expected, however, that submissions will be made to the UOA where there is the most appropriate
expertise to assess the body of work as a whole.
Unit of Assessment 8: Chemistry
76. The UOA includes all areas of experimental and theoretical chemistry, including appropriate
areas of pharmacy, chemical engineering and materials science, where the research is primarily
concerned with chemical aspects rather than clinical or engineering.
77. The sub-panel expects submissions in this UOA from all areas of chemistry, as defined above,
and expects that the majority of the research activity submitted will have made a direct contribution
to the UOA as characterised in the UOA descriptor. It recognises and welcomes, however, the
increasingly interdisciplinary nature of research, and expects that submissions may contain work that
contributes to this UOA and other disciplines, including those which have boundaries with this UOA,
17
such as UOA 5 (Biological Sciences), UOA 7 (Earth Systems and Environmental Sciences), UOA 9
(Physics) and other cognate disciplines.
Unit of Assessment 9: Physics
78. The UOA includes all areas of physics encompassing, but not limited to, theoretical,
computational and experimental studies of: quantum physics; atomic, molecular and optical physics;
plasma physics; fusion and energy; particle physics; nuclear physics; surface and interface physics;
condensed matter, materials and soft matter physics; biophysics; semiconductors, nanoscale
physics, lasers, optoelectronics and photonics; magnetism, superconductivity and quantum fluids;
fluid dynamics; statistical mechanics, chaotic and nonlinear systems; astronomy and astrophysics,
planetary and atmospheric physics; space physics; cosmology and relativity; medical physics;
applied physics; chemical physics; and instrumentation.
79. The sub-panel expects submissions in this UOA from all areas of physics, as defined above,
and expects that the majority of the research activity submitted will have made a direct contribution
to the UOA as characterised in the UOA descriptor. It recognises and welcomes, however, the
increasingly interdisciplinary nature of research, and expects that submissions may contain work that
contributes to this UOA and other cognate disciplines.
Unit of Assessment 10: Mathematical Sciences
80. The UOA includes pure and applied mathematics, statistics and operational research,
including the development and application of these areas in the study of biological, physical and
social sciences, commerce, engineering, finance, government, health, industry, information science,
medicine and elsewhere.
81. It therefore includes: algebra; analysis; category theory; combinatorics; complexity theory;
continuum mechanics and magnetohydrodynamics; differential equations; dynamical systems and
ergodic theory; environmental, financial, geophysical and industrial mathematics; geometry;
integrable systems; mathematical biology; mathematical logic; mathematical methods; mathematical
aspects of operational research, including optimisation and stochastic modelling; mathematical
physics; number theory; numerical analysis and scientific computing; operator theory and operator
algebras; probability; statistics such as experimental design, mathematical statistics, statistical
computing and contributions to data science; and applications such as biostatistics, environmental
and social statistics; topology. This list is necessarily incomplete, and any research in which the
primary contribution is mathematical may be considered in this UOA, including experimental,
theoretical or computational investigations related to mathematical or statistical models applied in
other subject areas.
82. The sub-panel expects submissions in this UOA from all areas of mathematical sciences, as
defined above, and expects that the majority of the research activity submitted will have made a
direct contribution to the UOA as characterised in the UOA descriptor. The sub-panel welcomes the
submission of innovative interdisciplinary research that incorporates mathematical, statistical or
operational research content, irrespective of the primary research focus of the medium in which the
18
output is disseminated. It also expects to receive some outputs on the history of mathematical
sciences when they incorporate insights into mathematics or statistics.
83. The sub-panel will receive outputs describing purely pedagogic research within higher
education and where advice is required from Sub-panel 23 (Education), outputs will be cross-
referred. Operational research that is focused on business and management should not normally be
submitted in this UOA.
Unit of Assessment 11: Computer Science and Informatics
84. The UOA includes the study and evaluation of methods for acquiring, storing, processing,
communicating and reasoning about information and interactivity in natural and engineered systems,
as characterised by the Association for Computing Machinery (ACM) Computing Classification
System, 2012 Revision https://www.acm.org/publications/class-2012. The sub-panel expects
submissions in this UOA to include contributions from ACM categories hardware, computer systems
organisation, networks, software and its engineering, theory of computation, mathematics of
computing, information systems, security and privacy, human-centered computing, computing
methodologies, and applied computing. The field is characterised by strong theoretical foundations
and systematic application of analysis, design, experimentation and evaluation.
85. The sub-panel expects that the majority of the research activity submitted will have made a
direct contribution to the UOA as characterised above, either by contributing new methods and
knowledge or through innovative applications of state-of-the-art methods to challenging problems in
other disciplines. Consequently, the sub-panel welcomes submissions containing interdisciplinary
outputs that make contributions to other areas as well as computer science and informatics, though
outputs that apply routine computational methods may be better returned elsewhere.
Unit of Assessment 12: Engineering
86. The UOA includes all areas of: aeronautical, mechanical and manufacturing engineering; bio-
engineering; chemical engineering; civil and construction engineering; electrical and electronic
engineering; metallurgy, materials science and engineering; and general engineering. It includes, but
is not limited to, research carried out in: additive manufacturing; aerodynamics; aerospace
engineering; amorphous materials and glasses; antennae and radar; architectural engineering;
artificial intelligence and its applications; automotive engineering; avionics; battery technology;
biochemical engineering; bioengineering; biomaterials; biomedical engineering; building engineering
and physics; ceramics; climate change; combustion; communications and networks; composites;
computational methods; computer and software engineering; computer vision; construction and
infrastructure; construction management; construction materials; control and systems; corrosion
engineering; cryptography; data engineering; digital manufacturing; dynamics; earthquake
engineering; electrical power systems, machines and drives; electromagnetics and its applications;
electrochemical engineering; electronic devices; electronic materials; electronic systems and circuits;
energy and power engineering; energy harvesting and scavenging; engineering biology; engineering
design; engineering ethics; engineering management; environmental engineering; extreme events;
19
fire engineering; fluid mechanics; fluid power; fluid structure interactions; fluidics; food process
engineering; fuel technology; functional materials; geomatics and surveying; geospatial engineering;
geotechnical engineering; health and safety; healthcare technologies; human factors and
ergonomics; hybrid materials; hydraulics and hydrology; information engineering; innovation
management and policy; instrumentation and measurement; intelligent and adaptive systems; life
cycle analysis; machine learning; manufacturing technology, processes and systems; maritime
engineering; materials engineering; materials for energy applications; materials characterisation;
materials modelling; materials processing; mechanics; mechatronics; metals; metrology;
microelectromechanical systems; mineral and mining engineering; modelling and simulation;
multimedia; music technology; nanoelectronics; nanomaterials; nanotechnology; natural materials;
naval architecture; new materials; non-destructive testing and structural health assessment; nuclear
engineering; offshore and coastal engineering; particle technology; photonics and its applications;
polymer and large area electronics; polymers; power electronics; process and product engineering;
product design and verification; project management; quantum technologies; radio frequency
techniques; railway engineering; recycling and green technologies; renewable energy; risk, reliability
and resilience; robotics and automation; sensors and actuators; signal and image processing; solar
cells and systems; solid mechanics; space engineering; speech and language technology; structural
dynamics; structural materials; structures; surfaces and interfaces; sustainability engineering;
systems engineering; systems modelling and identification; technology and operations management;
textiles; thermodynamics and heat transfer; tissue engineering; transportation engineering; tribology
and wear; turbo-machinery and propulsion; vibration and acoustics; water and waste water
engineering; wind engineering; and wireless networks. It also includes pedagogic research in
engineering.
87. The sub-panel expects submissions in this UOA from all areas of engineering, as defined
above, and expects that the majority of the research activity submitted will have made a direct
contribution to the UOA as characterised in the UOA descriptor. It recognises and welcomes,
however, the increasingly interdisciplinary nature of research in this area, and expects that
submissions may contain outputs that not only make contributions to this UOA and other cognate
disciplines but also to UOAs that extend beyond traditional cognate disciplines.
Main Panel C: UOAs 1324
Introduction
88. The sub-panels of Main Panel C cover a diverse range of content, disciplines and
methodologies. The sub-panels anticipate receiving research outputs, impact case studies, and
impact and environment templates which reflect that rich diversity, and have no pre-conceptions
about where excellent research will be found.
89. Each sub-panel expects to receive submissions whose primary research focus falls within the
stated remit of its UOA. Submitting units are encouraged to submit their strongest work, including
interdisciplinary work, in the UOA where it is most appropriate.
20
90. Criminology is a multi-disciplinary subject concerned with crime, criminals and criminal justice.
This could be submitted into various sub-panels. For the avoidance of doubt, it is recognised that
much criminological research may fall within the boundaries of Sub-panels 18 (Law), 20 (Social
Work and Social Policy) and 21 (Sociology). All three sub-panels welcome such work, which will be
assessed in accordance with the arrangements noted above, in particular making use of calibration
exercises, joint assessors and cross-referral as deemed appropriate by the sub-panels. Following
the assessment, the relevant sub-panels will review the health of UK criminological research, and
will report on this in a discrete section of the panel overview reports.
Unit of assessment descriptors and boundaries
Unit of Assessment 13: Architecture, Built Environment and Planning
91. Descriptor: The UOA covers all forms of historical, theoretical, technical, policy, applied and
practice-based research relevant to the planning, design, creation, functionality, use, conservation,
interpretation, assessment, management and governance of the built environment in both rural and
urban areas. This includes: architecture and related arts, building engineering, building surveying,
building sciences, climate change and disaster resilience, communities, construction, construction
management, economic development, environment, health and well-being, housing, landscape,
manufacture, natural resources and ecosystem services, real estate, regeneration, spatial analysis,
sustainability, transport, urban and regional planning and urbanism. It covers the social, economic,
legal, financial, environmental, technological, historic and cultural aspects of the built environment.
The UOA also covers any other research in which the built environment forms a major field for
application or provides the context for such research. It expects submissions in this UOA from a
broad range of disciplines, research methodologies and forms of output, across the spectrum of
fundamental, applied, pedagogical, policy and practice-based research. The submitted research may
span disciplinary and methodological boundaries.
92. Boundaries: The sub-panel expects submissions in this UOA from all areas outlined in the
UOA descriptor, but anticipates submissions that may span the boundaries between two or more
UOAs. Submitting units are encouraged to submit outputs that are of interdisciplinary nature, even if
the research is at the boundaries of the UOA.
Unit of Assessment 14: Geography and Environmental Studies
93. Descriptor: The UOA covers all aspects of research conceptual, methodological,
substantive and applied conducted within the disciplines of Geography and Environmental Studies,
as broadly defined, and contributing to their interdisciplinary collaborations. This research embraces
a wide range of enquiries into natural, environmental and human phenomena, and their
interrelationships in particular systems, contexts, periods and places, both in the UK and
internationally.
94. In Geography, submitted research may include work from all fields of physical and human
geography (for example, biogeography and ecosystem science; climatology; environmental
processes; environmental and climate change; forensic, geomorphology, glaciology, hydrology,
21
ocean and water science; Quaternary science; soil science; environmental geography, risks and
hazards; cultural, development, economic, health, heritage, historical, legal, political, population,
rural, social, transport, and urban geographies; geohumanities; and geographical information
sciences and Earth observation); work that combines any of these fields (for example, in socio-
ecological systems and natural resource governance); and work that uses a wide range of available
methods, from science-based to humanistic and participatory, including numerical, theoretical,
experimental, modelling, archival and field-based.
95. In Environmental Studies, submitted research may include work in any area of the field,
including those also present in environmental geography (for example, ecosystem services and
natural capital, environmental economics, politics, policy and practice, sustainable development);
some aspects of environmental science (for example, conservation, ecology, environmental pollution
and resource management); and environmental assessment and decision support systems.
96. Boundaries: Given the breadth of the subject matter of UOA 14, it is inevitable that the
exemplification above is not exhaustive and that there will be some overlaps with other UOAs,
located both in Main Panel C and in other main panels and that submission may include work that is
close to the boundaries of the UOA. In areas where there is significant overlap between UOA 14 and
another UOA, it is expected that whole submissions will be made in the UOA appropriate to the
academic context and research environment in which the research was undertaken, and with the
most appropriate range of expertise for the body of work as a whole.
Unit of Assessment 15: Archaeology
97. Descriptor: In Archaeology, submitted research may cover all fields of the subject for
example, including but not limited to: archaeological theory and historiography; archaeological
science and archaeological methods; the archaeology of human origins and evolution, and
prehistoric and historic societies worldwide; early civilisations (including classical archaeology and
related historical and textual studies), medieval and post-medieval to contemporary archaeology;
colonial, industrial and maritime archaeology; landscape and environmental archaeology;
archaeological aspects of heritage; heritage science; museum studies; archaeological conservation;
and forensic archaeology.
98. Boundaries: Submitting units are encouraged to submit their strongest work irrespective of
the form of output or the extent of its interdisciplinary nature, even if the research is at the
boundaries of the UOA. There could be overlaps with any UOA, particularly UOA 1 (Clinical
Medicine), UOA 5 (Biological Sciences) in Main Panel A; UOA 7 (Earth Systems and Environmental
Sciences) in Main Panel B; UOA 14 (Geography and Environmental Studies) and UOA 22
(Anthropology and Development Studies) in Main Panel C; and UOA 28 (History), UOA 29 (Classics)
and UOA 34 (Communication, Cultural and Media Studies, Library and Information Management) in
Main Panel D.
22
Unit of Assessment 16: Economics and Econometrics
99. Descriptor: The UOA is fully inclusive of all areas of economics and econometrics including
economic history. Research of all types empirical or theoretical, applied, or policy-focused will be
considered of equal standing.
100. Boundaries: Units are encouraged to submit their strongest work irrespective of the form of
output or the extent of its interdisciplinary nature, even if the research is at the boundaries of the
UOA. There could be overlaps with any UOA, including the other UOAs within Main Panel C,
particularly UOA 17 (Business and Management Studies). These parts of submissions will normally
be cross-referred, following advice from the cross-UOA members, to Sub-panel 17. In common with
any cross-referred work Sub-panel 16 (as the sub-panel for the UOA in which the work was
submitted for assessment) will retain responsibility for recommending the quality profile.
Unit of Assessment 17: Business and Management Studies
101. Descriptor: The UOA includes (but is not restricted to) the areas of: accounting; banking;
business analytics; business and industrial economics; business ethics; business history; consumer
behaviour; corporate governance; corporate social responsibility; critical management studies;
employment relations; entrepreneurship; finance; human resource management; information
systems management; innovation management; international business; leadership; management
education; management science; marketing; operations management; organisational psychology;
organisational studies; project management; public policy; public sector management; risk
management; service management; small firms; strategic management; supply chain management;
sustainability; technology management; third sector management; and any other field or sub-field
aligned to business and management.
102. Boundaries: Institutions are encouraged to submit their strongest work, irrespective of the
form of output. A variety of methodologies and interdisciplinary contributions is welcome. The sub-
panel anticipates that some of the work submitted in this UOA may overlap with the remits of UOA
10 (Mathematical Sciences), UOA 14 (Geography and Environmental Studies) and UOA 16
(Economics and Econometrics).
103. The main panel’s preferred approach is that the majority of work submitted in a UOA is
assessed by that sub-panel. However, significant aspects of submissions in UOA 17 (Business and
Management Studies) are expected to fall within the remit of UOA 16 (Economics and
Econometrics). These parts of submissions, following advice from the cross-UOA member(s) will
normally be cross-referred to Sub-panel 16. In common with any cross-referred work, Sub-panel 17
(as the sub-panel for the UOA in which the work was submitted for assessment) will retain
responsibility for recommending the quality profile.
23
Unit of Assessment 18: Law
104. Descriptor: The UOA includes all doctrinal, theoretical, empirical, comparative, critical,
historical or other studies of law and legal phenomena including criminology, and socio-legal studies.
The sub-panel would also expect research on legal education to be submitted in this UOA.
105. Boundaries: All areas of law as described above fall within the boundaries of the UOA. Law is
a hybrid, multi-disciplinary subject which draws on disciplines in both the social sciences and the
humanities. Research in law may intersect with or draw upon a variety of disciplines and
methodologies. The sub-panel has been constituted with a broad spread of relevant expertise to
ensure informed assessment of all submissions, and encourages units to submit their strongest work
including research which is at the boundaries of the UOA.
106. For the avoidance of doubt, it is recognised that much criminological research may fall within
the boundaries of Sub-panels 18 (Law), 20 (Social Work and Social Policy) and 21 (Sociology). All
three sub-panels welcome such work, which will be assessed in accordance with the arrangements
noted above, making use of calibration, joint assessors and cross-referral as deemed appropriate by
the sub-panels.
Unit of Assessment 19: Politics and International Studies
107. Descriptor: Politics and international studies has a broad spectrum, including: comparative
politics; area studies, international development, national and sub-national and grassroots politics;
studies of political institutions, public administration, policy and governance; the examination of
power, authority and legitimacy; political behaviour, political sociology and political economy; and
political theory and philosophy, including histories of political and international thought. It also
includes international relations theory; security studies including strategic, war and peace studies;
conflict research; international history; international political economy; and foreign policy analysis.
The sub-panel will welcome work from across this spectrum, including work that draws on a wide
range of theoretical approaches, among them feminist, postcolonial and queer perspectives, and on
diverse methods in political and international studies, among them quantitative, qualitative and
multimethod work as well as formal analysis and conceptual research. Outputs on pedagogic
research in politics and international studies will also be welcome.
108. Boundaries: The sub-panel expects to receive submissions in the UOA from all areas of the
discipline. It recognises that the boundaries of politics and international studies are not fixed and
welcomes work that is interdisciplinary, multidisciplinary and transdisciplinary in nature. The sub-
panel expects to read most of the material ‘in-house’. However, it will draw on the expertise of other
sub-panels where appropriate.
Unit of Assessment 20: Social Work and Social Policy
109. Descriptor: The UOA covers all forms of research in social work, social policy and
administration, and criminology/criminal justice policy, gerontology, and substantive issues in these
studies. Research includes, but is not restricted to:
24
a. theory, methodology, empirical research, reviews/syntheses, analyses of documents,
records and statistics, ethics and values, and pedagogy as these areas apply to social
work, social care, social policy, criminology and criminal justice policy, gerontology and
substantive issues in these areas of study
b. research that defines and seeks to understand social problems and their impact
c. comparative research and research into international institutions and all forms of welfare
politics, policy and practice and conceptions of crime and criminal justice
d. research that uses a range of disciplinary and interdisciplinary approaches including (but
not limited to): business and management, demography, development studies,
economics, education, geography, health studies, housing and urban studies, history,
law, philosophy, politics, public policy, psychology, social anthropology and sociology
e. policy-making processes, practice, governance and management, service design,
delivery and use, criminal justice design, and inter-professional relationships
f. research that includes links and co-production with, a variety of stakeholders (public
and private), professionals, volunteers, service users/participants and carers
g. research that analyses, evaluates and provides critical insights into the intersection of
the areas listed in ae with key dimensions of representation and identity particularly
marginal and/or excluded groups and communities for example: social class, gender,
sexuality, ‘race’/ethnicity, disability, age and migration/citizenship status.
110. Boundaries: Social work, social policy and administration, and criminology/criminal justice are
subjects closely related to a number of other disciplines within and outside the social sciences.
Political science, education, law, geography, social anthropology, developmental and social
psychology, are clearly areas the boundaries of which overlap with this UOA but certain types of
historical research and the ‘social aspects’ of, inter alia, environment studies, genetics/biomedicine,
and engineering are also areas from which the sub-panel could expect to receive work.
111. Work submitted to this sub-panel may overlap significantly with the remit of Sub-panel 21
(Sociology). It is anticipated that the use of joint assessors and cross-referral of parts of submissions
may be required in order to ensure an appropriate assessment process, in accordance with the
arrangements in Part 5, paragraphs 399 to 404.
112. For the avoidance of doubt, it is recognised that much criminological research may fall within
the boundaries of sub-panels, in particular Sub-panels 18 (Law), 20 (Social Work and Social Policy)
and 21 (Sociology). All three sub-panels welcome such work, which will be assessed in accordance
with the arrangements noted above, making use of calibration, joint assessors and cross-referral as
deemed appropriate by the sub-panels.
25
Unit of Assessment 21: Sociology
113. Descriptor: Sociology is a social science with a diversity of areas and approaches to the study
of social life and society. It includes empirical, critical and theoretical study of social structures,
power, cultures and everyday practices, including styles and material standards of living, opinions,
values and institutions. It includes analysis of and attention to social inequalities, divisions, justice
and solidarities at the micro, meso and macro levels. It covers all areas of social theory, historical
and comparative studies, and social research methodology, philosophy of social science, and
research on pedagogy in sociology. Sociology embraces a wide range of methodologies including
quantitative, qualitative and visual; and of all forms of data. The sub-panel also expects to consider
sociological research in such interdisciplinary fields as criminology and socio-legal studies, media
and cultural studies, social policy, gender and women’s studies, demography, socio-linguistics,
social psychology, psychosocial studies, social studies of science and technology, and lesbian, gay,
bisexual, transgender, queer and intersex studies.
114. The sub-panel expects submissions in this UOA from all fields of sociological enquiry
including, but not restricted to, research on cultures, economies and polities; class, race, ethnicity,
gender, sexuality, disability, and age, and their intersections; religion, education, health and
medicine, family, media, welfare institutions, and work and employment; environment, technology
and the digital; and climate change; the body, interpersonal and inter-group relations, violence;
urban and rural issues; language and social interaction; political sociology, public policy and social
movements; political economy, globalisation, development, migration and diaspora; comparative
studies of societies of all kinds, including work on transnational structures and agencies, Europe
world systems.
115. As in previous research assessment exercises, work in interdisciplinary women’s and gender
studies may be submitted in this UOA, or may be cross-referred by other sub-panels to Sub-panel
21 (Sociology).
116. Work submitted in this UOA may overlap significantly with the remit of UOA 20 (Social Work
and Social Policy). It is anticipated that the use of calibration, joint assessors and cross-referral of
parts of submissions may be required in order to ensure an appropriate assessment, in accordance
with the arrangements in Part 5, paragraphs 399 to 404.
117. For the avoidance of doubt, it is recognised that much criminological research may fall within
the boundaries of Sub-panels 18 (Law), 20 (Social Work and Social Policy) and 21 (Sociology). All
three sub-panels welcome such work, which will be assessed in accordance with the arrangements
noted above, making use of calibration, joint assessors and cross-referral as deemed appropriate by
the sub-panels.
118. Boundaries: Sub-panel 21 (Sociology) acknowledge that the UOA does not specify
boundaries as per many other sub-panels. This is balanced through the descriptor outlining
processes for managing submissions, to a greater detail than other sub-panels. The sub-panel
consider issues of boundaries to have been addressed.
26
Unit of Assessment 22: Anthropology and Development Studies
119. Descriptor: The UOA covers all aspects of research within Anthropology and Development
Studies, including research that is conceptual, theoretical, empirical, applied, strategic and practice-
based, and that draws on a broad range of methodologies that includes the qualitative, quantitative,
field-based, ethnographic, laboratory-based, experimental, participatory, evaluative, visual and
comparative.
120. Anthropology is understood to include the broad fields of biological anthropology,
palaeoanthropology, and social and cultural anthropology. Social and cultural anthropology includes,
but is not limited to, economic and political anthropology; kinship, gender and relatedness; religion;
cognition; medical anthropology; psychological anthropology; environment, conservation and
biodiversity; the anthropology of development; visual anthropology; ethnomusicology and
performance; material culture; and digital anthropology. Biological anthropology includes, but is not
limited to, human and non-human primate evolution and adaptation; palaeoanthropology; behaviour,
growth and development; health and disease; ecology; conservation; genetics; demography; and for
forensic applications.
121. Development Studies involves the analysis of global, national and local processes of change,
including social, economic, political, demographic, cultural, environmental and technological. These
may be studied from micro- to macro-scale and from local to global levels, with particular attention to
the relations between these. Attention is often paid to contexts characterised by poverty, inequalities,
environmental vulnerability and socio-political conflict and fragility. Research is often issue-driven
and involves critical interrogation of development and humanitarian theories, structures, processes,
policies and practices. Its focus can include, for example, poverty reduction, equalities,
empowerment, peace and reduction of violence, and food security. It can be multidisciplinary or
interdisciplinary and may combine social science with other disciplines.
Unit of Assessment 23: Education
122. Descriptor: All outputs in submissions under UOA 23 should have an educational focus or
orientation. Research in education is closely related to a range of other disciplines with which it
shares common interests, methods and approaches. This diversity of content and methodology
requires the sub-panel to be flexible in setting out the boundaries of work relevant to the REF.
123. The UOA may be broadly described as being concerned with research in the areas identified
in the following illustrative lists:
Research which addresses education systems, issues, processes, provision and outcomes
in relation to sectors, such as: early years, primary, secondary, further, higher, medical,
workplace, adult and continuing education. It also includes teacher, healthcare and other
forms of professional education, vocational education and training; and informal, community
and lifelong learning.
27
Research which addresses substantive areas, such as: curriculum, pedagogy, assessment,
language, teaching and learning; children, young people, student and adult learners; parents,
families and communities; culture, economy and society; teacher training, professionalism
and continuing professional development (CPD); special and inclusive education;
participation, rights and equity issues; technology-enhanced learning; education policy; the
organisation, governance, management, effectiveness and improvement of educational
institutions; education, training, workplaces, industry and the labour market; comparative,
international and development education.
Research which employs a range of theoretical frameworks and methodologies drawn from
disciplinary traditions, including, but not limited to: anthropology, applied linguistics,
economics, geography, history, humanities, mathematics, statistics, philosophy, political
science, psychology, science and sociology. Research in the field of education deploys a
range of qualitative and quantitative methodologies with structured, exploratory and
participatory research designs. These include, but are not limited to: surveys, experiments
and controlled trials; ethnography, interview and narrative enquiry; action research and case
study; evaluation research; critical theory and documentary analysis; analytic synthesis; and
systematic review.
124. The sub-panel accepts submissions in pedagogical research in higher education (whether or
not this is generated in education departments or similar units) and in professional education
(including healthcare), while recognising that such work may instead be submitted in another
relevant UOA. The sub-panel will consider submissions in counselling and neuroscience. However,
submissions in these areas may be referred to another sub-panel for advice.
Unit of Assessment 24: Sport and Exercise Sciences, Leisure and Tourism
125. Descriptor: Research in the UOA stems from the natural sciences, social sciences and the
humanities. The sub-panel expects to receive submissions from a wide range of disciplines and
subject areas that contribute to research in sport and exercise sciences, leisure and tourism. This
could include (in alphabetical order): adapted physical activity, anthropology, behaviour change,
biochemistry, biomechanics, business management and marketing, coaching, culture and media
studies, development studies, economics, education and pedagogy, engineering and technology,
festivals and events, geography, history, hospitality, law, medicine, molecular biology, motor learning
and control, nutrition, outdoor and adventure education, philosophy, physical education, physical
activity and health/public health, physiology, policy studies, politics, psychology, sociology, sports
injury and rehabilitation, and strength and conditioning. Research in sport and exercise sciences,
leisure and tourism is therefore derived from diverse disciplines and subject areas, and can also be
multidisciplinary and interdisciplinary.
126. Boundaries: The sub-panel expects submissions in this UOA of research of all types, and it
expects to consider research informed by a variety of research epistemologies, methodologies and
methods. The sub-panel will consider research defined as empirical, theoretical, strategic, applied,
or policy-focused as having equal standing.
28
Main Panel D: UOAs 2534
Introduction
127. The main panel is charged with identifying excellence in the rich diversity of research covered
by the UOAs described below. It welcomes all outputs arising from this research, in whatever genre,
medium or location, that can be demonstrated to meet the definition of research for the REF, as
outlined in Annex C of the ‘Guidance on submissions’ and that have entered the public domain
during the publication period. The sub-panels are committed to applying criteria and working
methods that reflect the distinctive character, methodologies and full breadth of these disciplines
(including interdisciplinary research), and that facilitate the formation of a balanced range of
judgements, without privileging or disadvantaging any particular form of research output, research
methodology or type of research environment.
128. The main panel and its sub-panels will operate according to the following principles:
Panels will assess submissions in the form that HEIs have chosen to present their research,
within the REF framework.
Panels will aim to identify excellence wherever they can find it.
Unit of assessment descriptors and boundaries
Unit of Assessment 25: Area Studies
129. Taking an inclusive view of Area Studies as a dynamic field, the sub-panel understands Area
Studies broadly to include the study of all regions of the world, across any period of time (ancient,
medieval and modern) however defined, and the communities associated with them. As well as in
terms of national territories, regions may be delineated in various ways, including by traditional
geographical designations (e.g. African Studies, American and Anglophone Studies (Canada and
the United States); Asian Studies including Central Asian, North East Asian Studies (including China
and Japan), South Asian and South East Asian Studies; Latin American and Caribbean Studies;
Australian, New Zealand and Pacific Studies; European Studies, including Russian and East
European Studies; Middle Eastern Studies (including Jewish and Islamic Studies)); by ecological
(e.g. Circumpolar Studies), geopolitical (e.g. Post-Soviet Studies) or institutional (e.g. European
Union Studies) criteria; or in terms of themes, processes or networks (e.g. Diaspora Studies, Post-
Colonial Studies, Gender Studies, Intercultural Studies etc.).
130. The sub-panel has expertise across the humanities and social sciences and welcomes work in
any language from any single-disciplinary, interdisciplinary, multidisciplinary, empirical or theoretical
perspective. It will assess submissions covering all, but not limited to, aspects of anthropology, law,
history, heritages, languages and linguistics, cultures, literatures, religions, philosophy, media,
society, economics, human geography, sociology, politics and international relations, and translation,
as well as inter-regional and globalisation studies. It welcomes ground-breaking or novel approaches
and seeks to reward innovation and excellence in both traditional and non-traditional formats,
29
including applied, practice-based and pedagogical research. The sub-panel comprises specialists in
humanities and social sciences and particularly welcomes work that crosses the arts and
humanities/social sciences boundary.
131. The sub-panel has expertise to assess a wide range of work and takes an inclusive view of the
subject areas within its scope. Given the broad range of its descriptor, it recognises that submissions
may be made in this UOA that include work in languages, literatures, cultures and societies falling
wholly or partially outside its members’ expertise. The sub-panel anticipates it will work closely with,
as appropriate, other sub-panels e.g. UOA 26 (Modern Languages and Linguistics) and UOA 19
(Politics and International Studies). The sub-panel is also mindful of the likely need to appoint further
panel members for the assessment stage of the REF.
132. The sub-panel recognises the overlapping boundaries in the sub-panel descriptors, and that
aspects of research are naturally interdisciplinary or multi-disciplinary or span the boundaries
between individual UOAs. The sub-panel welcomes the submission of research that may equally be
submitted to other sub-panels. Where judged necessary by the panels, expertise will be augmented
by additional assessors, or work will be cross-referred to relevant panels, according to the process
detailed in Part 5, paragraphs 399 to 404.
Unit of Assessment 26: Modern Languages and Linguistics
133. The UOA includes research on the languages, literatures, cultures and societies of all regions,
countries and communities where Celtic, Germanic, Romance or Slavonic languages or other
languages of Europe and Latin America are, or were, used. This includes areas where European
languages have interacted with other cultures and languages. The UOA also includes all areas of
general, historical, theoretical, descriptive and applied linguistics, phonetics, and translation and
interpreting studies, regardless of the methodology used or the language to which the studies are
applied. The sub-panel will take a broad view of what constitutes modern language studies. This will
include, but not be limited to: literature and thought; cultural studies; theatre studies; film and media
studies; visual cultures; language studies; translation and interpreting studies; political, social and
historical studies; editorial scholarship, bibliography, textual criticism and theory and history of the
book; philosophy and critical theory; world literature and comparative literature; literature in relation
to the other arts; and applied, practice-based and pedagogical research, including translation and
creative writing. The sub-panel welcomes the submission of interdisciplinary research, including
work on language and literature in relation to science, medicine and technology, digital humanities,
or creative technologies, and will ensure that such work is assessed with appropriate expertise.
134. The sub-panel has expertise to assess a wide range of work and takes an inclusive view of the
subject areas within its scope. Given the broad range of its descriptor, it recognises that submissions
may be made in this UOA that include work on languages, literatures, cultures and societies falling
wholly or partially outside its members’ expertise. The sub-panel consequently expects some degree
of overlap with UOA 25 (Area Studies). It anticipates that the two sub-panels will work together
closely and as appropriate before and during the assessment period.
30
135. The sub-panel recognises the overlapping boundaries in the sub-panel descriptors, and that
aspects of research are naturally interdisciplinary or multidisciplinary or span the boundaries
between individual UOAs. The sub-panel welcomes the submission of research that may equally be
submitted to other sub-panels. Where judged necessary by the panels, expertise will be augmented
by additional assessors, or work will be cross-referred to relevant panels, according to the process
detailed in Part 5, paragraphs 399 to 404.
Unit of Assessment 27: English Language and Literature
136. The UOA includes all aspects of language studies, including all areas of linguistics and of
applied linguistics, with primary reference to any variety of English or Scots; the history of English or
Scots; Old Norse/Icelandic (language, literature and linguistic studies); English literature from the
early Middle Ages to the present day; North American literature; comparative literature; world
literatures in English; colonial and postcolonial literatures and languages; literatures translated into
English; women’s writing; creative writing and practice; life writing; children’s literature; creative non-
fiction and/or creative critical writing; critical and cultural theory; cultural history; gender and sexuality
studies; editorial scholarship, bibliography, textual criticism and theory, and history of the book; Irish
literature in English; Scottish literature in English and Scots; Welsh literature in English; and applied,
practice-based and pedagogical research in English.
137. The sub-panel will take a broad view of what constitutes English literature and language. It
recognises the overlapping boundaries in the sub-panel descriptors, and that aspects of research
are naturally interdisciplinary or multi-disciplinary or span the boundaries between individual UOAs.
The sub-panel aims to be inclusive, and welcomes the submission of research that may equally be
submitted to other sub-panels. This may include areas such as theatre and performance studies;
cultural studies; film, television and digital media studies; popular music; history; art history;
philosophy; the linguistics of languages other than those mentioned above; translation studies;
language and literature in relation to science, technology and medicine; medical and health
humanities; digital humanities; or creative technologies.
138. The sub-panel will ensure that such work is assessed with appropriate expertise. Where
judged necessary by the panels, expertise will be augmented by additional assessors, or work will
be cross-referred to relevant panels, according to the process detailed in Part 5, paragraphs 399 to
404. Specialist members of the relevant sub-panels will liaise to ensure that broad, cross-disciplinary
subject areas such as linguistics are appropriately assessed.
Unit of Assessment 28: History
139. The UOA includes all aspects of the study of the past.
140. The sub-panel expects submissions in this UOA from all areas of history. The following list (in
alphabetical order) is illustrative rather than exhaustive; it does not reflect any judgements about the
relative significance of the subject areas, nor does it specify ‘fields’: agricultural histories;
biographical histories; business histories; contemporary histories; cultural histories; ecclesiastical
histories; economic histories; histories of education; environmental histories; gender histories; global
31
histories; heritage; historiography; history and memory; histories of Britain, Ireland and Continental
Europe (late Roman to the present); histories of ethnicity; histories of ideas; histories of North
America, South America, Africa, Asia and Australasia; histories of race; histories of science,
technology and medicine; histories of sexuality; histories of the book; indigenous histories;
imperial/colonial histories; international histories; labour histories; local and regional histories;
manuscript studies; material culture; media histories; military histories; oral histories; political
histories; public histories; religious histories; rural histories; social histories; theories of history;
transnational histories; urban histories; women’s histories; and world histories.
141. All ancient history outputs will be automatically cross-referred to Sub-panel 29 (Classics).
Byzantine history outputs will also be cross-referred where it seems more appropriate for Sub-panel
29 to consider the output.
142. The sub-panel recognises the overlapping boundaries in the sub-panel descriptors, and that
aspects of research are naturally interdisciplinary or multi-disciplinary or span the boundaries
between individual UOAs. The sub-panel welcomes the submission of research that may equally be
submitted to other sub-panels. Where judged necessary by the panels, expertise will be augmented
by additional assessors, or work will be cross-referred to relevant panels, according to the process
detailed in Part 5, paragraphs 399 to 404.
Unit of Assessment 29: Classics
143. The UOA includes the language, literature, history, culture, art, archaeology and thought
(including ancient science and philosophy) of Greece and Rome from the earliest times to late
antiquity; Latin language and literature of the Middle Ages and subsequent periods; Ancient Egypt
and the ancient Near East, Byzantine studies; modern Greek language, literature, history and
culture; the classical tradition; and the reception of these periods and subjects.
144. Within the boundaries are the following: the Greek world from the Bronze Age to the fall of the
Byzantine Empire; the Roman world from the Bronze Age to late antiquity; Greek lands, including the
Diaspora, from the medieval period to the present; the philology and linguistics of Latin and Greek
and of related and neighbouring languages; comparative literature and such literature, literary
theory, philosophy, political thought, material culture, art, film, television, digital media, creative
practices, music, and such political, archaeological and other cultural activity as exploits in any way
the history or cultural products of the Greek, Roman and Byzantine worlds, including translation and
performance of classical works; the pedagogy associated with learning and teaching in the subjects
listed here.
145. The list above is illustrative rather than exhaustive. It does not reflect any judgements about
the relative significance of the subject areas, nor does it specify ‘fields’.
146. The sub-panel recognises the overlapping boundaries in the sub-panel descriptors, and that
aspects of research are naturally interdisciplinary or multi-disciplinary or span the boundaries
between individual UOAs. The sub-panel welcomes the submission of research that may equally be
32
submitted to other sub-panels. Where judged necessary by the panels, expertise will be augmented
by additional assessors, or work will be cross-referred to relevant panels, according to the process
detailed in Part 5, paragraphs 399 to 404.
Unit of Assessment 30: Philosophy
147. The UOA includes all areas and styles of, and approaches to, philosophy. The subpanel
expects to receive submissions from all areas of philosophy, and considers the following subjects
(listed alphabetically), among others, to be within the remit of the UOA: 19th to 21st century
European philosophy including phenomenology, existentialism, critical theory, hermeneutics, and
deconstruction; aesthetics; applied philosophy; environmental philosophy; epistemology; ethics,
including applied ethics and meta-ethics; feminist philosophy; history of philosophy including ancient,
medieval, modern and recent; logic; metaphysics; non-Western philosophy; philosophy and history
of mathematics; philosophy and history of science, technology and medicine; philosophy of
education; philosophy of language; philosophy of law; philosophy of mind; philosophy of race;
philosophy of religion; political and social philosophy. The areas mentioned are illustrative rather
than exhaustive, and do not reflect any judgement about the relative significance of the subject
areas.
148. Because philosophy engages with conceptual and foundational issues raised by other
disciplines, it spans boundaries with a number of other UOAs, including but not limited to all the
other UOAs within Main Panel D and the following UOAs within other main panels: UOA 1 (Clinical
Medicine), for example philosophy of biological sciences; UOA 2 (Public Health, Health Services and
Primary Care), for example medical ethics; UOA 4 (Psychology, Psychiatry and Neuroscience), for
example cognitive science; UOA 5 (Biological Sciences), for example philosophy of biological
sciences; UOA 9 (Physics), for example the philosophy of physics; UOA 10 (Mathematical
Sciences), for example mathematical logic; UOA 16 (Economics and Econometrics), for example
social choice theory and game theory; UOA 18 (Law), for example jurisprudence; UOA 19 (Politics
and International Studies), for example political theory; UOA 21 (Sociology), for example social
theory.
149. In accord with the Philosophy sub-panel’s aim to be inclusive, it covers all types of applied
philosophy relating to practical issues both within and outside academia. The remit also covers work
concerned with philosophical questions raised by other disciplines, for example work concerned with
the foundations, methods, epistemic status, or interpretation of findings or theories in the other
disciplines. The sub-panel may consider that work that merely references philosophical ideas without
engaging with them philosophically will have its excellence best assessed by another sub-panel, and
will consider cross-referral accordingly.
150. The sub-panel recognises the overlapping boundaries in the sub-panel descriptors, and that
aspects of research are naturally interdisciplinary or multi-disciplinary or span the boundaries
between individual UOAs. The sub-panel welcomes the submission of research that may equally be
submitted to other sub-panels. Where judged necessary by the panels, expertise will be augmented
33
by additional assessors, or work will be cross-referred to relevant panels, according to the process
detailed in Part 5, paragraphs 399 to 404.
Unit of Assessment 31: Theology and Religious Studies
151. The UOA encompasses all research in theology and religion, and is inclusive of all disciplinary
approaches adopted in the field, including philosophical, theological, historical, philological, literary,
phenomenological, psychological, sociological and anthropological methodologies. It is also inclusive
of research into the nature of theology and/or religious studies as disciplines, and of the
methodologies they employ.
152. It encompasses the study and interpretation of religious institutions, movements, texts, laws,
practices, ethics, beliefs, symbols, media, social relations, material objects, spaces and flows, both
historical and contemporary in local and/or global contexts. It includes all religious traditions,
spiritualities and sacralised forms of commitment and their expression in different cultural media
for example film, art, music and literature, in whatever genre or medium. The study of varieties of
secularism, secularity and non-belief which reference religion explicitly or implicitly is also included. It
also covers work concerned with theological and religious questions raised by other disciplines,
including being able to review work relating to the context, assumptions and content of religious
education in different settings.
153. Theology and Religious Studies is an inherently multi- and cross-disciplinary subject, and
religion intersects with many other aspects of society, politics and culture. In recognition of this, the
sub-panel will welcome submissions which overlap with the remit of other UOAs; or for which UOA
31 is not the only appropriate one; or from those undertaking relevant research in academic units not
classified as theology, divinity or religious studies; or from academic units which specialise in only
one area of the field.
154. Given the multi-disciplinary reach of UOA 31, it is anticipated that a substantial portion of
submissions received will overlap with other UOAs, for example with UOA 14 (Geography and
Environmental Studies), UOA 18 (Law), UOA 19 (Politics and International Studies), UOA 21
(Sociology), UOA 22 (Anthropology and Development Studies), UOA 23 (Education) and the UOAs
within Main Panel D. Sub-panel 31 contains considerable linguistic, methodological and cross-
disciplinary expertise, but will apply the arrangements set out in Part 5, paragraphs 399 to 404,
where expertise needs to be augmented. Sub-panel 31 continues to welcome innovative and cross-
disciplinary approaches to the study of religion as well as more traditional methods.
155. The sub-panel recognises the overlapping boundaries in the sub-panel descriptors, and that
aspects of research are naturally interdisciplinary or multi-disciplinary or span the boundaries
between individual UOAs. The sub-panel welcomes the submission of research that may equally be
submitted to other sub-panels. Where judged necessary by the panels, expertise will be augmented
by additional assessors, or work will be cross-referred to relevant panels, according to the process
detailed in Part 5, paragraphs 399 to 404.
34
Unit of Assessment 32: Art and Design: History, Practice and Theory
156. The sub-panel will assess research from all aspects of the history, theory and practice of art
and design, and disciplines where these relate to visual, material and spatial cultures. The sub-panel
will consider outputs, in whatever genre or medium, that meet the definition of research (as outlined
in ‘Guidance on submissions’, Annex C). The sub-panel acknowledges and welcomes a diverse
range of methods and approaches to research, and therefore adopts an inclusive definition of its
remit.
157. It is anticipated that outputs will span a range of texts, edited publications and creative
practices, as well as artefacts, events and curatorial outputs. The sub-panel expects to evaluate
research that encompasses analytical, applied, ethnographical, experimental, historical,
pedagogical, scientific, technological and theoretical approaches to the widest domains of the
history, theory and practice of art and design, and covers the broadest understanding of the subject
disciplines within any cultural, geographical or historical context.
158. The sub-panel is committed to applying criteria and working methods that are appropriate to all
submitting units, whatever their size or structure, without privileging any particular form of research
output or environment.
159. The sub-panel recognises the overlapping boundaries in the sub-panel descriptors, and that
aspects of research are naturally interdisciplinary or multi-disciplinary or span the boundaries
between individual UOAs. The sub-panel welcomes the submission of research that may equally be
submitted to other sub-panels. Where judged necessary by the panels, expertise will be augmented
by additional assessors, or work will be cross-referred to relevant panels, according to the process
detailed in Part 5, paragraphs 399 to 404.
Unit of Assessment 33: Music, Drama, Dance, Performing Arts, Film and Screen
Studies
160. The sub-panel will assess research from all areas of music, drama, dance, theatre,
performance, live and sonic art, film, television and screen studies. It anticipates that outputs will
span a range of artefacts, creative practices, curatorial outputs, edited publications, recordings and
writings. The sub-panel expects to evaluate research that encompasses analytical, applied, critical,
ethnographical, historical, interdisciplinary, pedagogical, practice-based, scientific, technological and
theoretical approaches to all of the subject areas indicated above. It covers the broadest
understanding of the subject disciplines within, between and across any cultural, geographical or
historical contexts.
161. The sub-panel recognises the overlapping boundaries in the sub-panel descriptors, and that
aspects of research are naturally interdisciplinary or multi-disciplinary or span the boundaries
between individual UOAs. The sub-panel welcomes the submission of research that may equally be
submitted to other sub-panels. Where judged necessary by the panels, expertise will be augmented
by additional assessors, or work will be cross-referred to relevant panels, according to the process
detailed in Part 5, paragraphs 399 to 404.
35
Unit of Assessment 34: Communication, Cultural and Media Studies, Library and
Information Management
162. The sub-panel recognises the rich diversity of research across the Arts, Humanities and Social
Sciences in communication, cultural and media studies, library and information management, and
welcomes all outputs arising from this research, in whatever genre or medium, that can be
demonstrated to meet the definition of research for the REF (as outlined in ‘Guidance on
submissions’, Annex C). In setting out its remit, the sub-panel recognises that the UOA descriptor
covers two broad fields of research which are often distinct both organisationally and academically,
and welcomes submissions that reflect this. It also recognises that the activities covered by its remit,
even within its two broad fields of coverage, are often rooted in quite distinct research traditions or
infrastructures. It will assess research on its merits, with no penalty for research which is plainly
within a distinct tradition within the sub-panel’s remit. It will nonetheless welcome research which
seeks to engage with questions and concerns, such as the ‘information society’, heritage (both
cultural and museum aspects), networks or convergence, which may transcend field boundaries.
163. The UOA includes research that addresses or deploys theory, history, institutional, policy,
textual, critical and/or empirical analysis, or practice within communication, culture, media,
journalism, film, television and screen studies. Within UK higher education much, but not all, of this
work is likely to emanate from units or departments in communication studies, cultural studies,
media studies, journalism, or film and screen/television studies. This work will include research on
online and screen-based media (such as film, television, games and other digital forms), print media,
computer-mediated communication, digital infrastructure and platform studies focused on data and
society, diverse information and communication technologies, cultural policy, the creative industries
and popular culture, which will be variably titled and organised. The sub-panel will assess research
as defined above which addresses (but is not confined to): policy for regulation of culture and the
media and communication industries; the organisation, institutions, political economy and practice of
cultural production; media and cultural texts, forms and practices; media and cultural audiences,
consumption and reception; the role of changing technology, including emergent digital technologies,
in media production, content manipulation, distribution, access and participation. It is recognised that
this will include work which explores questions of power, identity and difference in relation to media,
communication and cultural studies which may sit at the intersections of (among others) gender and
sexuality studies, race and postcolonial studies, and disability studies.
164. The UOA also includes research concerned with the management of information and
knowledge in all formats, namely librarianship and information science, archives and records
management, and information systems. This concerns research on the generation, organisation,
dissemination and publication, exploitation, protection, and evaluation of information and knowledge,
and the impacts of such activities. It may include, for example, research that focuses on digital
humanities; digital participation; information behaviour and use; information ethics; information
literacy; information media; information policy; information retrieval; information security; information
seeking; the information society; knowledge management systems; preservation and conservation;
systems thinking; systems development; and the cultural, economic, ethical, historical, philosophical,
and societal aspects of the disciplines and their associated professions.
36
165. The sub-panel recognises the overlapping boundaries in the sub-panel descriptors, and that
aspects of research are naturally interdisciplinary or multi-disciplinary or span the boundaries
between individual UOAs. The sub-panel welcomes the submission of research that may equally be
submitted to other sub-panels. Where judged necessary by the panels, expertise will be augmented
by additional assessors, or work will be cross-referred to relevant panels, according to the process
detailed in Part 5, paragraphs 399 to 404.
37
Part 3: Assessment criteria
Section 1: Submissions
Interdisciplinary research
166. The REF main and sub-panels welcome the submission of interdisciplinary research, as
defined in paragraph 390, in any relevant UOA.
167. The arrangements for assessing interdisciplinary research including through the guidance
provided by the interdisciplinary advisers on the main and sub-panels, the interdisciplinary research
identifier, and the appointment of additional members and assessors are set out in the working
methods (see paragraphs 390 to 398).
168. The interdisciplinary research identifier for outputs allows HEIs to identify in the submission
those outputs which it considers to be interdisciplinary, and to draw this to the panels’ attention. This
is intended to give greater confidence to HEIs to submit interdisciplinary research, and will allow
panels, working with their IDR advisers, to consider the most appropriate means of assessing the
output. It is recognised that submissions may cover a broad range of disciplines, or may not have an
identifiable disciplinary focus. The sub-panels therefore encourage HEIs to identify in submissions all
outputs they consider meet the definition of interdisciplinary research (set out in paragraph 390)
where they wish to draw this to the panels’ attention. The sub-panels recognise that outputs flagged
as interdisciplinary may incorporate research crossing main panel areas, may span disciplines
across the sub-panels within a main panel, or may incorporate research areas covered within a sub-
panel particularly where UOAs cover a broad range of disciplines.
169. There will be no advantage or disadvantage in the assessment in identifying outputs as
interdisciplinary. The main and sub-panels will apply the standards of excellence defined by the
starred quality levels equally to research in interdisciplinary areas and to research within distinct
disciplines. The main and sub-panels consider that all such research is capable of displaying the
highest standards of quality.
Work on the boundaries between UOAs
170. The REF main and sub-panels recognise the diverse nature of the disciplines that they cover,
that UOAs do not have firm or rigidly definable boundaries, and that aspects of research span the
boundaries between individual UOAs, whether within a main panel or across main panels. They also
recognise that there are research areas which may be undertaken in a range of different contexts,
and some of these therefore occur in the descriptors of a number of UOAs. The main and sub-
panels welcome the submission of such research, in any relevant UOA.
171. The arrangements for assessing submissions that span UOA boundaries including the
cross-referral process are set out in the working methods (see paragraphs 399 to 404). The UOA
descriptors indicate where the panels might expect work submitted in their UOA to cross boundaries
with other UOAs, but recognise that there may be other overlaps.
38
172. Panels will assess, on an equal basis, submissions that reflect the work of administrative units
such as departments, and submissions that do not map neatly onto departmental or other
administrative structures within HEIs. In either case, institutions will not be penalised if submissions
contain some work that overlaps UOA boundaries. The main and sub-panels will apply the standards
of excellence defined by the starred quality levels equally to research that spans UOA boundaries
and to research solely within the remit of one UOA. The main and sub-panels consider that all such
research is capable of displaying the highest standards of quality.
Pedagogic research
Main Panel A criteria pedagogic research
173. It is expected that research on pedagogy or medical or veterinary education will be
submitted in UOA 23 (Education) and research on medical ethics will be submitted in UOA 30
(Philosophy). Research on the philosophical and ethical aspects of healthcare and on education
relevant to its disciplines may be submitted in UOA 3 (Allied Health Professions, Dentistry,
Nursing and Pharmacy) and any such outputs are expected to be assessed by that panel.
Main Panel B criteria pedagogic research
174. Research on pedagogy and educational issues within higher education that relate to the
disciplines covered by Main Panel B may be submitted in the UOA to which it relates rather than
to UOA 23 (Education). Such research will be assessed by the sub-panel for the UOA in which it
is submitted and where advice is required from Sub-panel 23 (Education), outputs will be cross-
referred.
175. Bodies of research into teaching in other education sectors or on general educational
issues should be submitted in UOA 23 (Education). Individual outputs on these issues received by
the sub-panels in Main Panel B will normally be cross-referred to Sub-panel 23 (Education) as
appropriate.
Main Panel C criteria pedagogic research
176. Research on pedagogy and educational issues within higher education that relate to the
disciplines covered by Main Panel C may be submitted in the UOA to which it relates or in UOA 23
(Education), as deemed appropriate by submitting HEIs.
Main Panel D criteria pedagogic research
39
177. Research on pedagogy and educational issues that relate to the disciplines covered by
Main Panel D may be submitted in the UOA to which it relates or in UOA 23 (Education), as
deemed appropriate by submitting HEIs. Main Panel D anticipates that individual sub-panels will
normally assess such research where it relates to education in the sub-panel’s discipline area.
Multiple submissions
178. ‘Guidance on submissions’ (paragraphs 73 to 77) sets out the arrangements whereby
institutions may exceptionally, and only with prior permission from the REF director, make more than
one submission (multiple submissions) in the same UOA. These exceptions are:
a. Where an institution involved in a joint submission wishes to make an additional
individual submission in the same UOA.
b. Multiple submissions to Sub-panel 26 (Modern Languages and Linguistics) will be
permitted where one submission is in Celtic Studies and the other in Modern Languages
and Linguistics. This has been agreed in recognition of the special cultural significance
of Celtic Studies in parts of the UK, and the particular legal status of the Welsh language
in Wales. For other types of multiple submissions in UOA 26, see paragraph 185.
c. Where HEIs merged after 1 July 2018, they may seek permission to make two separate
submissions in all of the UOAs in which they wish to submit, if, for example, they
anticipate difficulty in achieving academic cohesion between the merger date and the
submission date. Permission is unlikely to be granted to such HEIs to make separate
submissions only in some of the UOAs in which they wish to submit. In the event that
HEIs merged prior to 1 July 2018, the merged HEI should normally make one
submission only to each UOA.
d. Where a sub-panel considers there is a case for multiple submissions in its UOA, given
the nature of the disciplines covered, the institution may request a multiple submission.
The panels’ expectations are set out in paragraphs 180 to 186.
179. Each submission will be awarded a single overall quality profile. Where a single submission
includes distinct organisational units or areas of research and where the REF sub-panel considers it
appropriate, the sub-panel will provide feedback to the head of institution relating to the distinct units
or areas of research.
Main Panel A supplementary criteria multiple submissions
40
180. Sub-panels in Main Panel A do not consider that there is a case for multiple submissions in
their UOAs, based on the nature of the disciplines covered, and do not expect to receive requests
for multiple submissions in these UOAs (other than for the reasons stated in paragraph 178 a.-c.).
181. Sub-panel 3 (Allied Health Professions, Dentistry, Nursing and Pharmacy) recognises that
institutions may wish to receive output sub-profiles for distinct areas covered in their submission
and will provide output sub-profiles against the following areas to the head of institution where
requested: Nursing, Dentistry, Pharmacy, Allied Health Professions and Biomedical Sciences. If
institutions wish to receive output sub-profiles, they should assign each output in their submission
to one of the above categories. Sub-profiles will not be provided for outputs which have not been
tagged. The normal expectation is that output sub-profiles will not be provided where there is a
small number of staff associated with those outputs (typically less than five FTE). Output sub-
profiles will be provided confidentially to the head of the institution concerned
Main Panel B supplementary criteria multiple submissions
182. Sub-panels in Main Panel B do not consider that there is a case for multiple submissions in
their UOAs, based on the nature of the disciplines covered, and do not expect to receive requests
for multiple submissions in these UOAs (other than for the reasons stated in paragraph 178 a.-c.).
183. Sub-panel 12 (Engineering) recognises that institutions may wish to receive output sub-
profiles for distinct areas of engineering covered in their submission and will provide output sub-
profiles against the following areas to the head of institution where requested: Aeronautical,
Mechanical and Manufacturing Engineering; Bio-engineering; Civil and Construction Engineering;
Chemical Engineering; Electrical and Electronic Engineering; General Engineering; Metallurgy
and Materials. If institutions wish to receive output sub-profiles, they should assign each output in
their submission to one of the above categories. Sub-profiles will not be provided for outputs
which have not been tagged. If a ‘combined’ sub-profile is required for all remaining outputs (as
different from the overall output profile) then institutions should assign all remaining outputs to
general engineering. The normal expectation is that output sub-profiles will not be provided where
there is a small number of staff associated with those outputs (typically less than five FTE).
Output sub-profiles will be provided confidentially to the head of the institution concerned.
Main Panel C supplementary criteria multiple submissions
184. In Sub-panel 22 (Anthropology and Development Studies) requests for multiple submissions
where institutions have separate Anthropology and Development Studies (or International
Development) departments would normally fulfil the criteria. The remainder of sub-panels in Main
Panel C do not consider that there is a case for multiple submissions in their UOAs, based on the
41
nature of the disciplines covered, and do not expect to receive requests for multiple submissions
(other than for the reasons stated in paragraph 178 a.-c.).
Main Panel D supplementary criteria multiple submissions
185. The following sub-panels in Main Panel D consider that there is a case, based on the nature
of the disciplines covered by their UOAs, for multiple submissions in these UOAs and would
expect to receive requests for:
Sub-panel 25 (Area Studies)
Sub-panel 26 (Modern Languages and Linguistics)
Sub-panel 32 (Art and Design: History, Practice and Theory)
Sub-panel 33 (Music, Drama, Dance, Performing Arts, Film and Screen Studies)
Sub-panel 34 (Communication, Cultural and Media Studies, Library and Information
Management)
186. Requests for multiple submissions may be made in other UOAs within Main Panel D but are
expected to be a rare occurrence. All such requests will be considered according to the criteria
and procedures in paragraphs 73 to 77 of ‘Guidance on submissions’.
Section 2: Staff
Independent researchers
187. ‘Guidance on submissions’ (paragraph 128) sets out that staff employed on ‘research only’
contracts must be independent researchers to meet the definition of Category A eligible. For the
purposes of the REF, an independent researcher is defined as an individual who undertakes self-
directed research, rather than carrying out another individual’s research programme.
188. Possible indicators of independence are listed below. Institutions should note that each
indicator may not individually demonstrate independence and, where appropriate, multiple factors
may need to be considered. Across all main panels, the following indicators would normally identify
research independence:
leading or acting as principal investigator or equivalent on an externally funded research
project
42
holding an independently won, competitively awarded fellowship where research
independence is a requirement. An illustrative, but not exhaustive, list of independent
fellowships can be found at www.ref.ac.uk, under Guidance
leading a research group or a substantial or specialised work package.
Main Panels C and D supplementary criteria independent researchers
189. In addition to the generic criteria specified in the ‘Guidance on submissions’, Main Panels C
and D also consider that the following attributes may generally indicate research independence in
their disciplines:
Being named as a Co-I on an externally funded research grant/award.
Having significant input into the design, conduct and interpretation of the research.
Section 3: Outputs
Criteria and level definitions
190. This section provides a descriptive account of how the sub-panels will interpret and apply the
generic criteria for assessing outputs and the starred quality levels. This descriptive account
expands on and complements the generic criteria and definitions in Annex A of ‘Guidance on
submissions’, but does not replace them.
191. Originality will be understood as the extent to which the output makes an important and
innovative contribution to understanding and knowledge in the field. Research outputs that
demonstrate originality may do one or more of the following: produce and interpret new empirical
findings or new material; engage with new and/or complex problems; develop innovative research
methods, methodologies and analytical techniques; show imaginative and creative scope; provide
new arguments and/or new forms of expression, formal innovations, interpretations and/or insights;
collect and engage with novel types of data; and/or advance theory or the analysis of doctrine, policy
or practice, and new forms of expression.
192. Significance will be understood as the extent to which the work has influenced, or has the
capacity to influence, knowledge and scholarly thought, or the development and understanding of
policy and/or practice.
193. Rigour will be understood as the extent to which the work demonstrates intellectual coherence
and integrity, and adopts robust and appropriate concepts, analyses, sources, theories and/or
methodologies.
43
194. The generic definitions of the starred quality levels in the overall quality profile in each of the
three sub-profiles outputs, impact and environment are in Annex A of ‘Guidance on
submissions’. The panels would like to emphasise that ‘world-leading’, ‘internationally’ and
‘nationally’ in this context refer to quality standards. They do not refer to the nature or geographical
scope of particular subjects, nor to the locus of research, nor its place of dissemination.
195. The main panels have set out below a descriptive account of the starred level definitions for
outputs, as they apply in each main panel. These are provided to inform their subject communities
about how the panels will apply the definitions in making their judgements. Variations in terminology
reflect disciplinary norms but do not indicate a difference in the quality standards themselves. These
descriptive accounts should be read alongside, but do not replace, the generic definitions.
Interdisciplinary research
196. Interdisciplinary outputs will be assessed against the generic criteria of originality, significance
and rigour. In assessing interdisciplinary outputs, the sub-panels will make use of guidance provided
by the Interdisciplinary Research Advisory Panel (IDAP) that originality and significance can be
identified in one, some or all of the constituent parts brought together in the work, or in their
integration; they do not need to be demonstrated across all contributing areas/fields. This guidance
will work in parallel with rather than replace the generic criteria of originality, significance and
rigour.
Main Panel A supplementary criteria level definitions
197. In assessing outputs, the sub-panels will look for evidence of the quality of the output in
terms of its originality, significance and rigour, and will apply the generic definitions of the starred
quality levels.
198. The sub-panels will look for evidence of some of the following types of characteristics of
quality, as appropriate to each of the starred quality levels:
scientific rigour and excellence, with regard to design, method, execution and analysis
significant addition to knowledge and to the conceptual framework of the field
actual significance of the research
the scale, challenge and logistical difficulty posed by the research
the logical coherence of argument
contribution to theory-building
significance of work to advance knowledge, skills, understanding and scholarship in
theory, practice, education, management and/or policy
44
applicability and significance to the relevant service users and research users
potential applicability for policy in, for example, health, healthcare, public health, food
security, animal health or welfare.
199. Unless there is sufficient evidence of at least one of the above, or the definition of research
used for the REF is not met, research outputs will be graded as ‘unclassified’.
200. The sub-panels welcome research practice that supports reproducible science and the
application of best practice. Examples include registered reports, pre-registration, publication of
data sets, experimental materials, analytic code, and use of reporting checklists for publication
purposes and those relating to the use of animals in research. These contribute to the evaluation
of rigour for submitted outputs. Replication studies may be submitted as outputs and will be
evaluated on the extent to which they contribute significant new knowledge, improved methods, or
advance theory or practice
1
.
201. The sub-panels will use citation information, where appropriate and available, as part of the
indication of academic significance to inform their assessment of output quality. Further details on
the use of citation data are provided in paragraphs 274 to 276.
Main Panel B supplementary criteria level definitions
202. In assessing outputs, the sub-panels will look for evidence of originality, significance and
rigour and apply the generic definitions of the starred quality levels as follows:
a. In assessing work as being four star (quality that is world-leading in terms of
originality, significance and rigour), sub-panels will expect to see evidence of, or
potential for, some of the following types of characteristics:
agenda-setting
research that is leading or at the forefront of the research area
great novelty in developing new thinking, new techniques or novel results
major influence on a research theme or field
1
Institutions may find it useful to refer to international guidelines such as the following:
ARRIVE https://www.nc3rs.org.uk/arrive-guidelines
CONSORT http://www.consort-statement.org/
PRISMA http://www.prisma-statement.org/
COPE http://publicationethics.org/
ICMJE http://www.icmje.org/
ITHENTICATE http://www.ithenticate.com/
45
developing new paradigms or fundamental new concepts for research
major changes in policy or practice
major influence on processes, production and management
major influence on user engagement.
b. In assessing work as being three star (quality that is internationally excellent in terms
of originality, significance and rigour but which falls short of the highest standards of
excellence), sub-panels will expect to see evidence of, or potential for, some of the
following types of characteristics:
makes important contributions to the field at an international standard
contributes important knowledge, ideas and techniques which are likely to have a
lasting influence, but are not necessarily leading to fundamental new concepts
significant changes to policies or practices
significant influence on processes, production and management
significant influence on user engagement.
c. In assessing work as being two star (quality that is recognised internationally in
terms of originality, significance and rigour), sub-panels will expect to see evidence
of, or potential for, some of the following types of characteristics:
provides useful knowledge and influences the field
involves incremental advances, which might include new knowledge which
conforms with existing ideas and paradigms, or model calculations using
established techniques or approaches
influence on policy or practice
influence on processes, production and management
influence on user engagement.
d. In assessing work as being one star (quality that is recognised nationally in terms of
originality, significance and rigour), sub-panels will expect to see evidence of, or
potential for, some of the following types of characteristics:
useful but unlikely to have more than a minor influence in the field
46
minor influence on policy or practice
minor influence on processes, production and management
minor influence on user engagement.
e. Research will be graded as ‘unclassified’ if it falls below the quality levels described
above or does not meet the definition of research used for the REF.
Main Panel C supplementary criteria level definitions
203. In assessing outputs, the sub-panels will look for evidence of originality, significance and
rigour, and apply the generic definitions of the starred quality levels as follows:
a. In assessing work as being four star (quality that is world-leading in terms of
originality, significance and rigour), sub-panels will expect to see some of the following
characteristics:
outstandingly novel in developing concepts, paradigms, techniques or outcomes
a primary or essential point of reference
a formative influence on the intellectual agenda
application of exceptionally rigorous research design and techniques of
investigation and analysis
generation of an exceptionally significant data set or research resource.
b. In assessing work as being three star (quality that is internationally excellent in terms
of originality, significance and rigour but which falls short of the highest standards of
excellence), sub-panels will expect to see some of the following characteristics:
novel in developing concepts, paradigms, techniques or outcomes
an important point of reference
contributing very important knowledge, ideas and techniques which are likely to
have a lasting influence on the intellectual agenda
application of robust and appropriate research design and techniques of
investigation and analysis
47
generation of a substantial data set or research resource.
c. In assessing work as being two star (quality that is recognised internationally in
terms of originality, significance and rigour), sub-panels will expect to see some of the
following characteristics:
providing important knowledge and the application of such knowledge
contributing to incremental and cumulative advances in knowledge
thorough and professional application of appropriate research design and
techniques of investigation and analysis.
d. In assessing work as being one star (quality that is recognised nationally in terms of
originality, significance and rigour), sub-panels will expect to see some of the
following characteristics:
providing useful knowledge, but unlikely to have more than a minor influence
an identifiable contribution to understanding, but largely framed by existing
paradigms or traditions of enquiry
competent application of appropriate research design and techniques of
investigation and analysis.
e. Research will be graded as ‘unclassified’ if it falls below the quality levels described
above or does not meet the definition of research used for the REF.
Main Panel D supplementary criteria level definitions
Interpretation of generic level definitions
204. The terms ‘world-leading’, ‘international’ and ‘national’ will be taken as quality benchmarks
within the generic definitions of the quality levels. They will relate to the actual, likely or deserved
influence of the work, whether in the UK, a particular country or region outside the UK, or on
international audiences more broadly. There will be no assumption of any necessary international
exposure in terms of publication or reception, or any necessary research content in terms of topic
or approach. Nor will there be an assumption that work published in a language other than
English or Welsh is necessarily of a quality that is or is not internationally benchmarked.
205. In assessing outputs, the sub-panels will look for evidence of originality, significance and
rigour and apply the generic definitions of the starred quality levels as follows:
48
a. In assessing work as being four star (quality that is world-leading in terms of
originality, significance and rigour), sub-panels will expect to see evidence of, or
potential for, some of the following types of characteristics across and possibly
beyond its area/field:
a primary or essential point of reference
of profound influence
instrumental in developing new thinking, practices, paradigms, policies or
audiences
a major expansion of the range and the depth of research and its application
outstandingly novel, innovative and/or creative.
b. In assessing work as being three star (quality that is internationally excellent in terms
of originality, significance and rigour but which falls short of the highest standards of
excellence), sub-panels will expect to see evidence of, or potential for, some of the
following types of characteristics across and possibly beyond its area/field:
an important point of reference
of considerable influence
a catalyst for, or important contribution to, new thinking, practices, paradigms,
policies or audiences
a significant expansion of the range and the depth of research and its application
significantly novel or innovative or creative.
c. In assessing work as being two star (quality that is recognised internationally in
terms of originality, significance and rigour), sub-panels will expect to see evidence
of, or potential for, some of the following types of characteristics across and possibly
beyond its area/field:
a recognised point of reference
of some influence
an incremental and cumulative advance on thinking, practices, paradigms,
policies or audiences
a useful contribution to the range or depth of research and its application.
49
d. In assessing work as being one star (quality that is recognised nationally in terms of
originality, significance and rigour), sub-panels will expect to see evidence of the
following characteristics within its area/field:
an identifiable contribution to understanding without advancing existing paradigms
of enquiry or practice
of minor influence.
e. A research output will be graded ‘unclassified’ if it is either:
below the quality threshold for one star; or
does not meet the definition of research used for the REF. (See ‘Guidance on
submissions, Annex C).
Output types
206. The main panels welcome all forms of research output that fulfil the eligibility criteria for the
REF (set out in Part 3, Section 2 of ‘Guidance on submissions’). All forms of output, in any language,
will be considered equitably, with no distinction being made between the type of research or form of
output submitted. The sub-panels will neither advantage nor disadvantage any type of research or
form of output. The main panels encourage submitting institutions to refer to the glossary of output
types for information on the categories under which outputs may be submitted for assessment (see
‘Guidance on submissions’, Annex K).
207. No sub-panel will use journal impact factors or any hierarchy of journals in their assessment of
outputs. No output will be privileged or disadvantaged on the basis of the publisher, where it is
published or the medium of its publication.
208. Reviews, textbooks and edited works (including editions of texts) and translations may be
included if they embody research as defined in Guidance on submissions’, Annex C. Editorships of
journals and other activities associated with the dissemination of research findings should not be
listed as an output on REF2.
209. Each submitted output needs to have a single classification selected from the list of eligible
output types (see ‘Guidance on submissions’, Annex K). The purpose of the classification is to assist
in the management of the collection and distribution of outputs, the allocation of outputs to expert
reviewers, and a post-submission analysis of types of outputs submitted. The sub-panel will assess
the research content of the material submitted regardless of the classification.
50
Main Panel D supplementary criteria output types
210. It is not unusual for an output submitted to the sub-panels in Main Panel D to encompass a
number of different output types, such as a ‘design’ output which includes a journal article and a
patent application; or an ‘artefact’ or prototype that has been the subject of an exhibition; or a data
set or database which includes critical insight or analysis; or a ‘composition’ that has also been a
performance or recording; or an exhibition where the research may be curatorial (or involve or
support co-curation) and/or evident in the development of the interpretative strategy, exhibition
text/narrative or catalogue. Submitting institutions should select a single output type, and the
panel will judge the research content of the material submitted regardless of the classification.
211. An additional classification of ‘Translation’ has been added to the list of output types, for the
submission of works of translation of literary or scholarly texts or other cultural documents that
constitute original, significant and rigorous research. Translations that meet the definition of
research will often exhibit a deep insight into the source material, while drawing on and reflecting
specialist knowledge of its historical, political, social and cultural contexts, and will also rely on a
detailed engagement with style in both the source and target languages. Research may, as a
result, be reflected in the critical apparatus associated with a translated text but will also be
inherent in the translation process itself. Such outputs will often contribute to the development and
maintenance of intellectual infrastructure of subjects and disciplines. They may demonstrate
research practice that is critical and/or creative, and may also serve as substantial interventions in
intellectual and cultural life in their own right.
212. It is also anticipated that commentaries will be submitted, if they embody research as
defined for the purposes of REF. Like some translations, commentaries often include research
that encompasses work with original manuscripts, textual criticism, the historical, political, social
and cultural context of a text, its history of reception and influence, and issues in its contemporary
interpretation. Commentaries should be submitted under the “output type” (listed in ‘Guidance on
submissions’, Annex K) which fits their form of publication, most commonly “authored book”.
213. For indicative guidance on what material to include in the submission, please refer to the
table of output types in Annex C and the summary of ‘Additional Information’ in Annex B.
Outputs with significant material in common
214. As stated in Guidance on submissions’ (paragraph 220), where two or more research outputs
within a submission include significant material in common, the sub-panels will assess each output
taking account of the common material only once. Where a sub-panel judges that they do not
contain sufficiently distinct material and should be treated as a single output, an unclassified score
would be given to the ‘missing’ output.
51
215. Where a submitted output includes significant material in common with an output submitted to
REF 2014, as stated in Guidance on submissions (paragraph 259), submissions should explain
how far the earlier work was revised to incorporate new material (maximum 100 words).
Co-authored outputs
216. As set out in the ‘Guidance on submissions’, for co-authored outputs:
The number of other authors will be required.
Where two or more co-authors or co-producers of an output are returned in different
submissions (whether from the same HEI or different HEIs), the output may be listed in any
or all of these submissions.
In exceptional cases, co-authored outputs may be submitted up to two times within a
submission. This applies only to submissions made to Main Panel D (paragraphs 233 to
235).
Where there are substantial pieces of co-authored work, reflecting large-scale or intensive
collaborative research within the same submitting unit, and a double-weighting request has
been submitted for the output, institutions may attribute the output to a maximum of two
members of staff returned within the same submission. This output may be counted as the
required minimum of one for each staff member. The inclusion of any reserve outputs in this
instance must be in accordance with the minima and maxima requirements where the panel
does not accept the request for double-weighting.
217. Institutions may only attribute co-authored outputs to individual members of staff who made a
substantial research contribution to the output. The main panels set out below their requirements for
information about the author’s contribution.
218. Where information is requested and a sub-panel judges that the staff member to whom the
output is attributed has not made a substantial research contribution to a co-authored output, the
sub-panel will grade that occurrence of the output as ‘unclassified’.
219. Information may also be requested through an audit to verify that an author made a substantial
research contribution to the output. Where this cannot be verified the output will be graded as
‘unclassified’.
220. Once a sub-panel has determined that each co-author’s contribution to the research content of
the output is distinct and substantial, it will assess the quality of the output as a whole, taking no
further regard of each individual co-author’s contribution.
52
Main Panel A supplementary criteria co-authored outputs
221. An output may only be submitted once in a UOA submission by any given HEI. Where co-
authors represent different UOAs within an HEI, the output can be submitted to each UOA. Where
co-authors come from different institutions the output can be submitted by each HEI.
Information required about the author’s contribution
222. No additional information is required in form REF2 about the author’s contribution to co-
authored outputs where either:
there are 15 authors or fewer; or
there are more than 15 authors but the submitted member of staff to whom the output is
attributed is identified as either lead or corresponding author (regardless of the number of
authors).
223. Whether first author, last author, alphabetical or some other order, Main Panel A considers
that the lead and corresponding authors should be easily identifiable within the submitted output.
Provided the submitted member of staff is clearly identifiable within the output as lead or
corresponding author, including any instances of where that role may be shared, no additional
information is required.
224. For each submitted co-authored output where there are more than 15 authors and where
the submitted member of staff is not identified as the lead or corresponding author, institutions are
required to affirm the substantial contribution to the research by the submitted member of staff.
This should be done by entering the following statements in REF2, including at least one element
from each of a and b:
a. The author made a substantial contribution either to the conception and design of the
study; or to the organisation of the conduct of the study; or to carrying out the study;
or to analysis and interpretation of study data.
And
b. The author helped draft the output; or critique the output for important intellectual
content.
225. Where the author contribution has been included in the output acknowledgements, this will
take precedence to the statement on co-authored outputs. Statements on author contribution will
be subject to audit.
53
Main Panel B supplementary criteria co-author contribution
226. An output may only be submitted once in a UOA submission by any given HEI. Where co-
authors represent different UOAs within an HEI, the output can be submitted to each UOA. Where
co-authors come from different institutions the output can be submitted by each HEI.
Additional requirement for information on co-authored outputs
Sub-panels 7, 8, 10, 11 and 12
227. The sub-panels do not require the submission of information about the individual co-
author’s contribution to a co-authored output and, if received, will take no account of such
statements.
Sub-panel 9 only
228. No additional information is required in form REF2 about the author’s contribution to co-
authored outputs where either:
there are 15 authors or fewer; or
there are more than 15 authors but the submitted member of staff to whom the output is
attributed is identified as either lead or corresponding author (regardless of the number of
authors).
229. Whether first author, last author, alphabetical or some other order, Sub-panel 9 considers
that the lead and corresponding authors should be easily identifiable within the submitted output.
Provided the submitted member of staff is clearly identifiable within the output as lead or
corresponding author, including any instances of where that role may be shared, no additional
information is required.
230. For each submitted co-authored output where there are more than 15 authors and where
the submitted member of staff is not identified as the lead or corresponding author, institutions are
required to affirm the substantial contribution to the research by the submitted member of staff.
This should be done by entering up to 100 words in which the author contribution is articulated.
231. Where the author contribution has been included in the output acknowledgements, this will
take precedence to the statement on co-authored outputs. Statements on author contribution may
be subject to audit.
54
Main Panel C supplementary criteria co-authored outputs
231. An output may only be submitted once in a UOA submission by any given HEI. Where co-
authors represent different UOAs within an HEI, the output can be submitted to each UOA. Where
co-authors come from different institutions the output can be submitted by each HEI.
Additional requirement for information on co-authored outputs
232. The sub-panels in Main Panel C do not require the submission of information about the
individual co-author’s contribution to a co-authored output and, if received, will take no account of
such statements. The sub-panels may seek to verify a contribution via audit in accordance with
paragraph 219.
Main Panel D supplementary criteria co-author contribution
233. Exceptionally, the sub-panels in Main Panel D will accept the inclusion of the same co-
authored output up to two times in a submission. This provision is in recognition of the constraints
to the size of the output pool as a result of a combination of factors, including publication patterns
in Main Panel D; that many submissions will be from small departments; and that a number of
UOAs in Main Panel D are comprised of a broad spectrum of sub-disciplines, many of which will
be in separate departments in submitting HEIs.
234. Such outputs should not account for more than five per cent of the outputs (or one output,
whichever is the greater) within a submission. Alternatively, where such outputs satisfy the
requirements for double-weighting, submitting HEIs should use the provision outlined in the final
bullet point in paragraph 216 instead no quota applies in that case. These two provisions cannot
be used in combination.
235. Consequently, a co-authored output can be submitted:
once as a single output; or
twice, attributed to two of the authors when it satisfies the criteria for double-
weighting (see paragraph 216); or
twice, attributed to two of the authors, within the quota of five per cent or one output
(whichever is the greater), when it does not meet the double-weighting criteria.
Additional requirement for information on co-authored outputs
236. The sub-panels in Main Panel D do not require the submission of information about the
individual co-author’s contribution to a co-authored output and, if received, will take no account of
such statements. The sub-panels may seek to verify a contribution via audit in accordance with
paragraph 219.
55
Double-weighted outputs
237. The main and sub-panels recognise that there will be cases where the scale of academic
investment in the research activity and/or the intellectual scope of the research output is
considerable. The main and sub-panels want to recognise and double-weight such outputs in the
assessment, so that they will count as two outputs both in a submission and in the calculation of the
outputs sub-profile. The main panels have set out below their expectations in relation to receiving
requests for double-weighting.
238. Institutions’ requests for double-weighting must be accompanied by a statement of up to 100
words explaining how the output satisfies the criteria.
239. As set out in the ‘Guidance on submissions’ (paragraphs 282 to 283), a reserve output may be
submitted where a request for double-weighting is made. The reserve output may be attributed to
any submitted member of staff, providing that it is in accordance with the minima and maxima
requirements for attributing outputs to staff.
240. Sub-panels will double-weight an output only if a request is made by the submitting institution,
and the case is accepted by the sub-panel. Sub-panels will not double-weight any output for which a
request has not been made by the institution.
241. Sub-panels will assess the claim for double-weighting separately from assessing the quality of
the output, and there is no presumption that double-weighted outputs will be assessed at higher-
quality grades. When assessing claims for double-weighting, the sub-panel will not privilege or
disadvantage any particular form of research or type of output.
Main Panels A and B supplementary criteria double-weighting
242. The sub-panels anticipate that they will double-weight outputs only where they derive from
substantial academic endeavour by the member of staff against whom the output is listed in the
submission. Such endeavour might be understood in terms of (but is not limited to) the ambition of
the project.
243. Considering the patterns of publication across Main Panel A and B’s areas of activity, the
sub-panels expect that such requests will occur only exceptionally. In particular, the sub-panels
anticipate that outputs published as journal articles and conference papers will not normally
embody work of this nature, and they therefore do not normally expect to receive requests for
double-weighting these types of outputs.
56
Main Panels C and D supplementary criteria double-weighting
244. The sub-panels strongly encourage submission of outputs of extended scale and scope for
consideration as double-weighted outputs.
245. The submission of a statement to evidence the claim for double-weighting is required and
should briefly outline the reasons for the request, addressing the characteristics below.
246. The sub-panels in Main Panels C and D have identified the following characteristics which
might apply (individually or in combination) to the research effort associated with a double-
weighted output:
the production of a longer-form output (e.g. book, long-duration creative work or multi-
component output) demonstrating sustained research effort
the generation of an extended or complex piece of research
the collection and analysis of a large body of material
the use of primary sources which were extended, complex or difficult to access
the presentation of a critical insight or argument which was dependent upon the
completion of a lengthy period of data collection or investigation of materials
the undertaking of a complex, extended and/or multi-layered process of creative
investigation (individual or collective)
the investigation of a given theme in considerable depth, from different perspectives,
and/or in relation to different contexts.
It is recognised that in some instances the characteristics listed in paragraph 246 may apply to
short-form outputs such as journal articles, book chapters and short-duration creative work and
justify the double-weighting of such items.
247. It is expected that most books, monographs, novels or longer-form outputs warrant double-
weighting, although claims will not automatically be accepted.
Additional information for outputs
248. The requirements for additional supporting information for each main panel are set out below.
A summary of additional information for outputs required by the main panels is set out in Annex B.
57
249. HEIs are instructed to ensure that additional supporting information for outputs is succinct,
verifiable, and externally referenced where appropriate. No other information should be included,
and sub-panels will take no account of any such information if submitted.
250. Where additional information is accepted, the information provided must not include citation
data or journal impact factors. Any panels that make use of citation data will be provided with the
data by the REF team. Sub-panels will take no account of any citation data provided directly by the
HEI. No sub-panel will use journal impact factors or any hierarchy of journals in their assessment of
outputs.
251. For research outputs in languages other than English (see ‘Guidance on submissions’,
paragraphs 285 to 287) submitted to all sub-panels, a short abstract (up to 100 words) in English
should be provided to describe the content and nature of the work. This abstract does not form part
of the assessment of the submitted output. See below for an exception applying to Sub-panel 26
(Modern Languages and Linguistics).
Main Panel A supplementary criteria additional information for outputs
Information about the research process and/or content
252. For non-text, or practice-based outputs (including patents, software and standards
documents), all sub-panels welcome the submission of a description in REF2 of the research
process and research content, where this is not evident within the output (maximum 300 words),
as described in ‘Guidance on submissions’ (paragraph 284.a.).
Factual information about significance
253. The sub-panels do not wish to receive additional information about the significance of
outputs (‘Guidance on submissions’, paragraph 284.b.) and, if received, will take no account of
any statement beyond those that have been requested by Main Panel A, as summarised in Annex
B.
Main Panel B supplementary criteria additional information for outputs
Information about the research process and/or content (Sub-panels 7 to 12)
254. For non-text, or practice-based outputs (including patents, software and standards
documents) all sub-panels require the submission of a description of the research process and
content, where this is not evident within the output (maximum 300 words).
58
255. For reviews, sub-panels welcome the identification of the original research or new insights
reported, to assist with the assessment of research quality (maximum 300 words).
Factual information about significance Sub-panels 11 and 12 only
256. Sub-panels 11 and 12 consider that the nature of their disciplines is such that the
significance of an output may not be fully evident within the output itself. They therefore invite
factual information to be provided (maximum 100 words) that could include, for example,
additional evidence about how an output has gained recognition, impacted the state of the art, led
to further developments, or has been applied.
257. HEIs are instructed to ensure that such evidence is succinct, verifiable, and externally
referenced where appropriate. Where claims are made relating to the industrial significance of the
output, the name and contact details of a senior industrialist must be given to allow verification of
claims. Information provided should not comprise a synopsis of the output, a volunteered opinion
as to the quality of the output or citation data, and information provided that is of this nature will be
disregarded. It is expected that, in most cases, sufficient information will be provided in
significantly fewer words than the 100-word limit.
Allocation of outputs for assessment
258. The following sub-panels request information to assist in allocating outputs to
appropriate readers during the assessment phase in 2021. The information will not be used for
any other purpose:
Sub-panels 7, 11 and 12 will provide subject-specific taxonomies. Submitting units will be
asked to identify the topics relevant to submitted outputs using the chosen taxonomy.
Sub-panel 10 will request up to two keywords which will categorise the topic(s) covered by
the output. The keywords may be taken from the MSC2020 taxonomy where appropriate
or may be provided by the submitting UoA. Submitting units are asked to be as consistent
as possible in use of their chosen keywords within their submission.
Main Panel C supplementary criteria additional information for outputs
Information about the research process and/or content
259. For any submitted outputs where the research content and/or process is not evident from
the output, such as non-text outputs or teaching materials, submissions should include a
statement which identifies the research questions, methodology and means of dissemination
(maximum 300 words).
59
Submission of practice-based outputs
260. To ensure that practice-based outputs are assessed on an equal basis with other outputs,
submissions should include an explanatory presentation of the building, design or intervention in
an easily-handled paper-based format (for example, a PDF which could include photographs,
figures or diagrams) sufficient to allow the panel both to understand the output without visiting it,
and to make a judgement of its research contribution.
261. For software and data sets, a full written description should be provided in a paper-based
format in order to avoid accessibility problems, including details of how and where the data set or
software can be accessed. Such access should preserve the anonymity of the reviewer.
262. Where the form of an output makes this essential, the paper-based submission may be
supplemented by limited visual material in an accessible format such as a video file.
Main Panel D supplementary criteria additional information for outputs
Submission of research outputs to Main Panel D
263. The table attached in Annex C indicates the classification of output types and Main Panel
D’s guidance on the content of submissions relating to each output type.
264. In all cases where the role of the researcher, or the research process, is not evident within
the submitted output, submitting units are strongly encouraged to submit a statement of up to 300
words.
265. For outputs where the role of the researcher or the research process is not evident in the
submitted output, submitting units have the following options in choosing how best to present the
output (irrespective of the classification):
As a single item, with a 300-word supporting statement.
As a multi-component output, with a 300-word supporting statement.
As a single item, supported by contextual information (previously called a ‘portfolio’). Both
the item and the contextual information may include moving image, sonic, visual or other
digital media or written text, as appropriate, to enable the panel to access the research
dimensions of the work and to assess its significance, originality and rigour. The 300-word
statement should be used to indicate what is the output and what is the contextual
information.
60
As a multi-component output, supported by contextual information (previously called a
‘portfolio’). Both the output and the contextual information may include moving image,
sonic, visual or other digital media or written text, as appropriate, to enable the panel to
access the research dimensions of the work and to assess its significance, originality and
rigour. The 300-word statement should be used to indicate what is the output and what is
the contextual information.
266. The entirety of the material submitted (the output and the 300-word statement where
provided) should provide the panel with coherent evidence of the research dimensions of the work
in terms of:
the research process the question and/or issues being explored, the process of
discovery, methods and/or methodologies, the creative and/or intellectual context or
literature review upon which the work draws, or challenges or critiques
the research insights the findings, discoveries or creative outcomes of that process
the dissemination how and where the insights or discoveries were ‘effectively shared.
This needs to satisfy the REF requirements around the dates at which work first entered
the public domain (‘Guidance on submissions’, paragraph 205.b.). The principle that no
output will be privileged or disadvantaged on the basis of the publisher, where it is
published or the medium of its publication (paragraph 217), will also apply in relation to the
broad range of modes through which practice outputs enter the public domain.
267. The bullet points above are derived from the REF definition of research as set out in the
‘Guidance on submissions’, Annex C. They are intended to assist HEIs by providing a flexible
framework for the succinct and coherent presentation of the output. The objective is to enable
panels to assess the originality, significance and rigour of the research. The sub-panels will ignore
any additional material that includes evaluative commentary on the perceived quality of a
research output.
268. The format for the presentation of outputs (irrespective of their classification) is flexible,
within the requirement to limit the format either to an electronic submission which is submitted via
the REF submission system either as a URL, Digital Object Identifier (DOI), or by uploading a
PDF; or as physical material which is sent to the REF team, and which may include
digital/electronic material on a media storage device e.g. USB, CD. An individual output cannot be
submitted both electronically via the REF submission system and as a physical output.
269. There will be many outputs that will meet the REF definition of research as “a process of
investigation, leading to new insights effectively shared” without the need for additional
information, and these may include examples of creative practice. Where the research process is
not self-evident, the guidance in paragraphs 264 to 267 should be followed.
61
Statement on research contribution
270. Sub-panels expect to receive anthologies, edited books, special issues of journals and
curatorial projects where the researcher has made a demonstrable contribution to the research
published (in addition to any chapter published in the same work). Where such a research
contribution is part or all of the output to be assessed, the whole work should be submitted.
Submitting units may provide a statement (of up to 300 words) to clarify the nature of the
individual’s research contribution.
Statement on rationale for grouping short items
271. Substantial dictionary or encyclopaedia entries and groups of short items including for
example portfolios of creative writing, or related critical works (where such work embodies
research as defined for the purposes of the REF in ‘Guidance on submissions’), may be submitted
as a single output, along with an explanation of the rationale for grouping such items (of up to 300
words).
Information for outputs in languages other than English
272. For research outputs in languages other than English (‘Guidance on submissions’,
paragraphs 285 to 287), a short abstract (of up to 100 words) in English should be provided to
describe the content and nature of the work. This abstract does not form part of the assessment
of the submitted output. This requirement is waived for outputs submitted in UOA 26 (Modern
Languages and Linguistics) if the output is produced in any of the languages within the remit of
that UOA: that is, all Celtic, Slavonic, Germanic and Romance languages.
Allocation of outputs for assessment
273. The following sub-panels request information to assist in allocating outputs to
appropriate readers during the assessment phase in 2021. The information will not be used for
any other purpose:
Sub-panels 26, 27, 28, 29, 33 and 34 will provide subject-specific taxonomies. Submitting
units will be asked to identify the topics relevant to submitted outputs using the chosen
taxonomy.
Citation data
274. The main panels set out below which of the sub-panels will use citation data. Where sub-
panels use citation data, the following criteria apply:
62
a. Where available and appropriate, citation data will be considered as an indicator of the
academic significance of the research output. This will only be one element to inform
peer-review judgements about the quality of the output, and will not be used as a
primary tool in the assessment.
b. The absence of citation data for an output will not be taken to mean an absence of
academic significance.
c. The sub-panels recognise that the citation count is sometimes, but not always, a reliable
indicator. They are also aware that such data may not always be available, and the level
of citations can vary across disciplines and across UOAs. Sub-panels will be mindful
that citation data may be an unreliable indicator for some forms of output (for example,
relating to applied research), and the limitations of such data for outputs in languages
other than English and for recent outputs. Sub-panels will take due regard of the
potential equalities implications of using citation data, as outlined in the ‘Equality briefing
for REF panels’ (2018/05).
d. Sub-panels will use citation data only where provided by the REF team, and will not
refer to any additional sources of bibliometric analysis, including in particular journal
impact factors and other journal rankings.
275. Those panels using citation data will do so within the framework set out in ‘Guidance on
submissions’ (paragraphs 288 to 292). Panels will continue to rely on expert review as the primary
means of assessing outputs, in order to reach rounded judgements about the full range of
assessment criteria (‘originality, significance and rigour’). They will also recognise the significance of
outputs beyond academia wherever appropriate, and will assess all outputs on an equal basis,
regardless of whether or not citation data is available for them.
276. The panels using citation data will receive guidance from the Forum for Responsible Research
Metrics to ensure that they are used appropriately.
Main Panel A supplementary criteria citation data
277. All sub-panels in Main Panel A will use citation data, where appropriate and available, as a
potential indicator of academic significance to inform the assessment of output quality.
Main Panel B supplementary criteria citation data
278. Sub-panels 7, 8, 9 and 11 acknowledge that citation data are widely used and consider that
they are well understood in the disciplines covered in their UOAs. These sub-panels will receive
63
citation data, where available, and may make use of the data as part of the indication of academic
significance to inform their assessment of output quality.
279. Sub-panels 10 and 12 believe that citation data in their disciplines cannot be used to
provide sufficient added value to inform the assessment of output quality. They therefore will not
receive nor make use of citation data.
Main Panel C supplementary criteria citation data
280. Sub-panel 16 (Economics and Econometrics) will receive citation data, where available, and
will make use of the data supplied by the REF team where it is considered appropriate as an
additional piece of supplementary evidence to support the initial assessment of outputs, not as a
determining factor. Sub-panel 16 will take account of the well-known limitations of citations,
including equality, diversity and inclusion issues.
281. The remaining sub-panels within Main Panel C will neither receive nor make use of citation
data.
Main Panel D supplementary criteria citation data
282. The sub-panels in Main Panel D will neither receive nor make use of citation data.
Section 4: Impact
Introduction
283. This section should be read alongside ‘Guidance on submissions’ Part 3, Section 3, which sets
out the generic definition of impact for the REF, the requirements for submitting impact case studies,
the associated eligibility guidelines, and the generic assessment criteria and level definitions. The
sub-panels will assess impact in accordance with this framework.
284. The main and sub-panels have determined that no one model or relationship will be
considered intrinsically preferable, and each impact case study will be assessed on its own merits.
285. In drawing up their assessment criteria and the advice to submitting institutions, the main
panels strongly advise institutions that the guidance provided here, particularly regarding examples
of impacts and evidence and/or indicators for those impacts, should not be read as exhaustive,
prescriptive or limiting either for institutions or panels. They also recognise that the examples
provided in Table 1 (Annex A) may fit under headings other than those to which they have been
64
presented. The main panels wish to encourage the submission of a broad range of types of impact,
as evidence of the strength and diversity of the impact of research across all disciplines, and
anticipate that extremely strong impact case studies will be submitted which do not relate to any of
the examples provided in the guidance. The examples are offered to assist institutions, not to
constrain them.
286. The panels also acknowledge that there are multiple and diverse pathways through which
research achieves impact. Impact may be the result of individual or collective research (or a
combination of these) within or between a range of organisations, within HE and beyond, including
collaboration beyond the UK. The associated impact may be achieved by a variety of possible
models: from individuals, to inter-institutional groups, to groups including both academic and non-
academic participants. The relationship between research and impact can be indirect or non-linear.
The impact of research may be foreseen or unforeseen. It can emerge as an end product, but can
also be demonstrated during the research process. Impact takes place through a wide variety of
mechanisms. It may effect change or enrichment for local, national or international communities,
groups or individuals. Consequently, public engagement may be an important feature of many case
studies, as the mechanism by which the impact claimed has been achieved.
Impact criteria
287. The sub-panels will assess the ‘reach and significance’ of impacts on the economy, society
and/or culture that were underpinned by excellent research conducted in the submitted unit,
according to the generic criteria and level definitions. This section provides a descriptive account of
how the sub-panels will interpret and apply the generic criteria for assessing impact.
288. Reach will be understood as the extent and/or diversity of the beneficiaries of the impact, as
relevant to the nature of the impact. Reach will be assessed in terms of the extent to which the
potential constituencies, number or groups of beneficiaries have been reached; it will not be
assessed in purely geographic terms, nor in terms of absolute numbers of beneficiaries. The criteria
will be applied wherever the impact occurred, regardless of geography or location, and whether in
the UK or abroad.
289. Significance will be understood as the degree to which the impact has enabled, enriched,
influenced, informed or changed the performance, policies, practices, products, services,
understanding, awareness or wellbeing of the beneficiaries.
290. The sub-panels will make an overall judgement about the reach and significance of impacts,
rather than assessing each criterion separately. While case studies need to demonstrate both reach
and significance, the balance between them may vary at all quality levels. The sub-panels will
exercise their judgement without privileging or disadvantaging either reach or significance.
291. HEIs may submit case studies describing impacts at any stage of development or maturity.
However, the assessment will be solely on the impact achieved during the assessment period,
65
regardless of its stage of maturity. No account will be taken of anticipated or future potential impact,
nor of impact that occurred outside the assessment period (1 August 2013 to 31 July 2020).
Continued impact case studies
292. As set out in the ‘Guidance on submissions’ (paragraphs 314 to 317), case studies continued
from examples submitted in 2014 will be eligible for submission in REF 2021. All impact case studies
submitted in REF 2021 must meet the same eligibility criteria, including the length of the window for
underpinning research (1 January 2000 to 31 December 2020) and the assessment period (1 August
2013 to 31 July 2020) for the impact described. The main panels set out below their expectations in
relation to receiving continued case studies in the assessment.
293. The panels encourage submitting units to submit their strongest case studies irrespective of
whether they are new examples or represent continuing impact from those submitted in REF 2014.
Main Panel A supplementary criteria continued case studies
294. Main Panel A will assess each case study on merit and wishes to receive information on
how any continued case study relates to that submitted in REF 2014. Panel members will have
access to the REF 2014 database
2
and may refer to this to understand the context of the 2021
case study.
Main Panels B, C and D supplementary criteria continued case studies
295. The sub-panels will assess each case study on merit and do not wish to receive information
on how any continued case study relates to that submitted to REF 2014. If any such information is
provided, the sub-panels will not take it into account during the assessment process.
Range of impacts
296. The main panels welcome case studies that describe any type(s) of impact which fulfil the
definition of impact for REF (see ‘Guidance on submissions’, paragraphs 297 to 302). They
acknowledge that impact may take many forms and occur in a wide range of spheres. They welcome
case studies which describe impacts that have provided benefits to one or more areas of the
economy, society, culture, public policy and services, health, production, environment, international
development or quality of life. The panels will also welcome impacts that describe changes or
benefits resulting from research that leads to a decision not to undertake a particular course of
2
https://impact.ref.ac.uk.
66
action. Sub-panels recognise the value of co-produced impact and expect institutions to ensure case
studies of this nature clearly acknowledge the work of partner organisations.
297. Impacts can be manifested in a wide variety of ways including, but not limited to: the many
types of beneficiary (individuals, organisations, communities, industry, regions and other entities);
impacts on products, processes, behaviours, policies, practices and understanding; and avoidance
of harm or the waste of resources in the widest sense. Impact of any type may be local, regional,
national or international, in any part of the world.
298. Research may underpin impact which provides benefits in more than one area. An impact
case study may therefore describe more than one type of impact arising from such bodies of work;
for example, a new drug can generate both health and economic impact, a new energy technology
can generate both environmental and economic impact, and a new exhibition or performance can
generate cultural, economic and social benefits.
299. Engaging the public with the submitting unit’s research (for example, through citizen science,
patient and public involvement in health, or through public and community engagement), is an
activity that may lead to impact. Sub-panels will welcome, and assess equitably, case studies
describing impacts achieved through public engagement, either as the main impact described or as
one facet of a wider range of impacts. Panels expect that case studies based on public engagement
will demonstrate both reach (for example, through audience or participant figures and demographics)
and significance, and will take both into account when assessing the impacts. Examples of impacts
arising from public engagement can be found as part of Table 1 (Annex A).
300. Examples are provided in Table 1 (Annex A) as a guide to the range of potential impacts that
may be eligible as case studies. The list is not exhaustive or exclusive, and does not rank examples
in any way. In making use of this to assist with the preparation of submissions, HEIs should note
that:
a. The list of types and examples of impacts is not intended to be exhaustive, and some
examples are relevant to more than one type of impact. Sub-panels wish to encourage
HEIs to submit case studies describing any impacts that meet the generic definition in
‘Guidance on submissions’, paragraphs 297 to 302.
b. HEIs are not required to align submitted case studies specifically with the particular
types of impact defined in the list.
301. Impacts on or through teaching within and beyond the submitting institution may be submitted,
and examples are included in Table 1 (Annex A). Where impact on teaching within the submitting
unit’s own institution is included in a case study, sub-panels will give consideration to the following:
the reach of the impact, that is the extent or diversity of the communities affected by the
change to teaching practice
67
the significance of the impact, that is, the extent to which teaching practice was enriched,
influenced or changed at the organisation(s) involved and/or the extent to which individuals
experiencing changed teaching practice were enriched, influenced or changed.
302. Sub-panels expect that impact on teaching within the submitting unit’s own institution may
most convincingly form a component of a wider case study that also includes impacts beyond the
institution.
303. The panels acknowledge that there may be impacts arising from research which take forms
such as holding public or private bodies to account or subjecting proposed changes in society, public
policy, business practices, and so on to public scrutiny. Such holding to account or public scrutiny
may have had the effect of a proposed change not taking place; there may be circumstances in
which this of itself is claimed as an impact. There may also be examples of research findings having
been communicated to, but not necessarily acted upon, by the intended audience, but which
nevertheless make a contribution to critical public debate around policy, social or business issues.
The panels also recognise that research findings may generate critique or dissent, which itself leads
to impact(s). For example, research may find that a government approach to a particular social,
health, food-/biosecurity or economic issue is not delivering its objectives, which leads to the
approach being questioned or modified.
Evidence of impact
304. Each case study must provide a clear and coherent narrative that includes an account of who
or what audiences, constituencies, groups, organisations, places, publics, sectors and so on, have
benefited, been influenced, or acted upon. The sub-panels will use their expert judgement regarding
the integrity, coherence and clarity of the narrative of each case study, but will expect the key claims
made in the narrative to be supported by evidence and indicators.
305. In assessing impact case studies, sub-panels will consider both the evidence linking excellent
research and bodies of work within the submitting unit to the impact(s) claimed, recognising that this
relationship can be indirect or non-linear, and the evidence of the reach and significance of the
impact. Within their narrative account in the case study, submitting units should provide the
indicators and evidence most appropriate to support the impact(s) claimed. Where using quantitative
indicators, institutions should follow the guidance on their standardised presentation, available at
www.ref.ac.uk, under Guidance.
306. Submitting units should focus on providing evidence of the impacts achieved, as distinct from
evidence of dissemination and uptake, in order to demonstrate both the reach and significance of the
impact(s) claimed (see paragraph 290). For example, attendance figures at an event may illustrate
the pathway to a change in understanding or awareness and provide an indication of the reach of
the impact. However, on their own, they would not serve as evidence of the significance of the
impact, which might be demonstrated, for example, through participant feedback or critical reviews.
68
307. Submitting units should ensure that, so far as possible, any evidence cited is independently
verifiable. Verifiable sources for key evidence and indicators should be provided in Section 5 of the
impact case study template and the relevant evidence provided to the REF team as set out in the
‘Guidance on submissions’ Part 3, Section 3.
308. The main panels recognise that some evidence in case studies may be of a confidential or
sensitive nature. The arrangements for submitting and assessing case studies that include such
material are set out in Guidance on submissions’, paragraphs 307 to 308.
309. The panels anticipate that impact case studies will refer to a wide range of types of evidence,
including qualitative, quantitative and tangible or material evidence, as appropriate. Individual case
studies may draw on a variety of forms of evidence and indicators. The panels do not wish to pre-
judge forms of evidence. They encourage submitting units to use evidence most appropriate to the
impact claimed. A diversity of evidence is welcome, and no type of evidence is inherently preferred
over another. Cited evidence should provide a convincing and verifiable link between the
underpinning research or bodies of work and the impact claimed, as well as convincing and
verifiable evidence of the reach and significance of the impact.
310. Testimonials should draw on statements of fact and relate specifically to the impact(s) claimed.
There may be occasions where opinion-based testimonials are appropriate (for example, where the
impact is on public understanding of an issue). Where such testimonials are cited as evidence in
impact case studies, sub-panels will consider the extent to which the testimonial citation evidences
the significance of the claims. Sub-panels recognise the varying degrees to which evidence and
indicator information may be available to HEIs. Where testimony is cited, it should be made clear
whether the source is a participant in the process of impact delivery (and the degree to which this is
the case), or is a reporter on the process.
311. Where corroborating evidence is reviewed as a consequence of audit, it will be used solely to
verify the claims made about the impact. Additional information included in such evidence will not be
used to supplement or strengthen the impact case study narrative.
312. The examples in Table 1 (Annex A) provide a guide to potential types of evidence or indicators
that may be most relevant to each of the broad areas of impact described in Table 1. However,
institutions should note that:
This is not intended to be exhaustive.
Some indicators may be relevant to more than one type of impact.
Sub-panels will consider any relevant, verifiable evidence.
69
Underpinning research
313. Sub-panels need to be assured that the impact claimed is based on research at least
equivalent to two star, as defined in ‘Guidance on submissions’, Annex A. Submitting units are
required to identify the underpinning research and provide in Section 3 up to six key references to
research produced by the submitting unit in the period 1 January 2000 to 31 December 2020 that
underpins the impact described in the case study. The sub-panels will not expect each referenced
item to meet the quality threshold, but will wish to be satisfied that the research as a whole was of at
least two-star quality.
314. Underpinning research may be a body of work produced over a number of years, within the
REF timeframe (1 January 2000 to 31 December 2020) or may be the output(s) of a particular
project. It may be produced by one or more individuals. Underpinning research outputs may include
the full range of types listed in the output glossary (‘Guidance on submissions’, Annex K) and are not
limited to printed academic work. They may include, but are not limited to: new materials, devices,
images, artefacts, products and buildings; confidential or technical reports; intellectual property,
whether in patents or other forms; performances, exhibits or events; and work published in non-print
media. All forms of output cited as underpinning research will be considered on an equal basis, with
no distinction being made between the types of output referenced.
315. Provided the sub-panel is satisfied that the quality threshold has been met, the quality of the
underpinning research will not be taken into consideration as part of the assessment of the reach
and significance of the claimed impact.
316. Underpinning research referenced in a case study may also be included in a submission as an
output (listed in REF2), without disadvantage. In these situations, the assessment of the impact case
study will have no bearing on the assessment of the quality of the output. The assessment of the
quality of the output may inform the assessment of the case study, only in terms of assuring the
threshold for underpinning research quality.
317. The institution submitting a case study must have produced research which has made a
distinct and material contribution to the impact described in the case study. Sub-panels will expect to
see clear narrative evidence of this in the case study. The panels recognise that several groups,
institutions or organisations may have made distinct research contributions to a given impact, and
strongly advise submitting institutions to ensure that both their own contribution is specified clearly
and that the contributions of others are acknowledged.
318. There will be many cases where a researcher has moved to a different institution during the
period in which a body of research underpinning a case study was produced. Where this is the case,
the submitting institution should make clear that the research undertaken during the period the
researcher spent at that institution made a material and distinct contribution to the impact claimed.
70
Main Panels A and B supplementary criteria indicators of quality for underpinning
research
319. Case studies must include references to one or more key research outputs that identify the
research produced by the submitted unit that underpinned the impact, and must provide evidence
of the quality of the research. Case studies should include references to any REF-eligible
output(s) as defined in the output glossary that will best enable the panels to determine that the
two-star threshold has been met. They should include additional indicators, as appropriate, of the
quality of the underpinning research. The sub-panels will use the information provided in case
studies, and may review research and outputs referenced in Section 3, in order to be assured that
the quality threshold has been met.
Main Panels C and D supplementary criteria indicators of quality for underpinning
research
320. Submitting units must ensure that each case study fulfils the threshold criterion on research
quality (see ‘Guidance on submissions’, paragraph 319.b). A sample of the research should be
cited that is sufficient to identify clearly the body of work, or individual project that underpins the
claimed impact.
321. Main Panels C and D wish to emphasise that the term ‘underpinning’ does not imply a
specific temporal or directly causal relationship to the associated impact and recognise that in
some cases the research associated with the impact may be carried out at the same time as the
impact, and that the nature of the relationship between the research and the impact may be
complex and non-linear.
322. Sub-panels do not expect to read the underpinning research output(s) as a matter of course
to establish that the threshold has been met. The submitting institution should aim, where
possible, to provide evidence of this quality level. Some of the indicators of such quality might be
(but are not restricted to):
research outputs which have been through a rigorous peer-review process
evidence of peer-reviewed funding
reviews of outputs from authoritative sources
prizes or awards made to individual research outputs cited in the underpinning research
evidence that an output is a reference point for further research beyond the original
institution.
71
323. Not all indicators of quality will apply to all forms of research output. If no such indicators are
available then the sub-panel will review the item in question to satisfy itself that this meets the
quality threshold.
Preparing impact case studies
324. The sub-panels recommend that institutions refer to the following list of characteristics when
preparing case studies:
All the material required to make a judgement should be included in the case study template
(REF3) no further reading should be required. URLs should only be included for the
purpose of verifying or corroborating claims made in the submission. Panels will not follow
URLs to access additional evidence or information to supplement the submission.
There should be a clear definition of the beneficiaries, and what has changed as a result of
the research.
The narrative should be coherent, clearly explaining the relationship between the
researchers, the underpinning research, the impact, and the nature of the changes or
benefits arising (noting that narratives differ according to the areas of impact claimed).
Indicators used should be relevant, contextualised and precise in support of the case study,
and the evidence should be verifiable, focused and concise.
There should be a brief explanation of what is original or distinctive about the research
insights that contributed to the impact.
Specific and appropriate sources of corroborating information, independent of the submitting
HEI, should be supplied. Extracts from corroborating statements may be included within the
case studies, where appropriate.
Where the research was carried out in collaboration with other HEIs, or was part of a wider
body of research, this should be acknowledged and the specific contribution to the impact of
the submitting unit’s research clearly described. In such cases, units (whether within or
across HEIs) may provide common descriptions of the impact arising, where they so wish.
72
Section 5: Environment
Environment criteria
325. The sub-panels will assess the environment according to the generic criteria and level
definitions in ‘Guidance on submissions’, Annex A. The main panels have set out below how the
criteria will be understood by their sub-panels.
326. Vitality will be understood as the extent to which a unit supports a thriving and inclusive
research culture for all staff and research students, that is based on a clearly articulated strategy for
research and enabling its impact, is engaged with the national and international research and user
communities and is able to attract excellent postgraduate and postdoctoral researchers.
327. Sustainability will be understood as the extent to which the research environment ensures
the future health, diversity, wellbeing and wider contribution of the unit and the discipline(s),
including investment in people and in infrastructure.
328. In assessing the environment element of submissions, panels will assess vitality and
sustainability in terms appropriate to the scale and diversity of the research activity the submitting
unit supports, and as appropriate for its subject area(s). They will assess vitality and sustainability in
terms of both the research environment within the submitting unit, and its participation in and
contribution to its subject discipline, academic community and wider society.
329. In forming the environment sub-profiles, the sub-panels will attach weightings to each of the
four sections of the unit-level environment template (REF5b), as set out below:
unit context and structure, research and impact strategy
people
income, infrastructure and facilities
collaboration and contribution to the research base, economy and society.
The assessment will take account of the environment data as stated in paragraphs 362 to 363 and
the information provided in the institutional-level statement (REF5a) as stated in paragraph 333.
Main Panels A, B and C supplementary criteria section weightings for the environment
template (REF5b)
330. The sub-panels in Main Panels A, B and C will attach equal weighting to each of the four
sections of the unit-level environment template (REF5b).
73
Main Panel D supplementary criteria section weightings for the environment template
(REF5b)
331. In view of the primary role that people play as the key resource in the arts and humanities,
the sub-panels in Main Panel D will attach differential weight to each of the components of the
environment template as follows:
Unit context and structure, research and impact strategy
25%
People
30%
Income, infrastructure and facilities
20%
Collaboration and contribution to the research base,
economy and society
25%
Institutional-level environment template (REF5a)
332. The following information is required in the institutional-level environment statement (REF5a):
a. Context and mission: an overview of the size, structure and mission of the institution.
b. Strategy: the institution’s strategy for research and enabling impact (including integrity,
open research, considerations of equality and diversity, and structures to support
interdisciplinary research, where applicable) in the assessment period and for the next
five-year period.
c. People: the institution’s staffing strategy, support and training of research students, and
building on the information provided in codes of practice, evidence about how equality
and diversity in research careers is supported and promoted across the institution.
d. Income, infrastructure and facilities: the institutional-level resources and facilities
available to support research. This should include mechanisms for supporting the
reproducibility of research as appropriate to the research focus of the HEI, and to
facilitate its impact.
333. The sub-panels will use the information provided in the institutional-level statement to inform
and contextualise their assessment of the relevant sections of the unit-level template. The
institutional-level statement will not be separately assessed or separately scored by the sub-panels.
Units should not repeat material covered in REF5a in REF5b and should cross-refer between the
statements, where appropriate.
74
334. As set out in ‘Guidance on submissions’ (paragraph 362), the REF5a statements will be
reviewed by a pilot assessment panel. The pilot exercise will consider the standalone assessment of
a discrete institutional-level environment element in future exercises, and will run concurrently to the
REF 2021 assessment.
335. Small and specialist institutions that will make a submission in one UOA only are encouraged
to submit a REF5a statement but are not required to do so. Where an HEI does not provide a REF5a
statement, the pilot panel will review the submitted REF5b template. In such cases, institutions
should ensure that sufficient information is provided in the REF5b template about the institution’s
context and should be guided by the supplementary guidance to be provided by the pilot
assessment panel in summer 2019. Additionally, where there is any distinction between the research
and impact strategies, policies, facilities and resources between the institution and the submitting
unit, this should be clearly identified in the REF5b template.
Unit-level environment template (REF5b)
336. The main panels believe that excellent research can be undertaken in a wide variety of
research structures and environments, and outstanding impacts achieved from within a wide variety
of research contexts and resulting from a wide diversity of approaches. The main panels consider
that the health of the disciplines represented within the sub-panels is well served by this variety. The
main panels have no pre-formed view of the ideal size or organisational structure for a research
environment, or of the ideal context or approach for enabling impact, and will judge each submission
on its merits, contextualised appropriately to the nature of institution. Panels will assess, on an equal
basis, submissions that reflect the work of administrative units such as departments, and
submissions that do not map neatly onto departmental or other administrative structures within HEIs.
337. Given that there is no expectation that the environment element of submissions relates to a
single coherent organisational unit, submissions should explain any distinct groups or units covered,
particularly where discrete organisational units form part of a single submission.
338. Neither the existence of research groups, nor their absence, is, in itself, considered significant
by the sub-panels.
Use of indicators
339. As indicated in paragraph 362, all sub-panels will receive information on doctoral degrees
awarded and research income. In the main panel supplementary criteria set out for the different
sections of the REF5b template, the panels indicate where additional data should be provided in the
template.
340. Further quantitative indicators may be included in REF5b, where appropriate, to support
claims made in the narrative. In identifying additional indicators for inclusion, submitting units are
strongly advised to refer to the advice and examples based on work carried out by the Forum for
Responsible Research Metrics, available at www.ref.ac.uk, under Guidance. However, the main
75
panels wish to make it clear that the examples should not be regarded as mandatory nor a ‘check-
list’ of additional requirements.
341. In assessing the environment template, sub-panels will take a holistic view of each section,
taking into account both the narrative and any supporting evidence provided. Where appropriate,
panels will consider data in the context of size and type of institution in order to enable judgements
to be made on an equitable basis.
Template requirements
342. The main and sub-panels expect to see the unit’s approach to equality and diversity reflected
and evidenced as appropriate throughout the template below.
343. All the material required to make a judgement should be included in the environment template
no further reading should be required. URLs should only be included for the purpose of verifying or
corroborating claims made in the submission. Panels will not follow URLs to access additional
evidence or information to supplement the submission.
344. Submitting institutions are reminded that the word limits for REF5a and REF5b (set out in
Annex F of the ‘Guidance on submissions’) are upper limits, not a minimum requirement.
345. The following information is requested for each of the sections in the environment template
(REF5b):
REF5b, Section 1: Unit context and structure, research and impact strategy
346. This section should provide evidence of the achievement of strategic aims for research and
impact during the assessment period, and details of future strategic aims and goals for research and
impact; how these relate to the structure of the unit, and how they will be taken forward. Note that
there is no expectation that this section refers to a single department or coherent organisational unit.
Evidence may include (but is not limited to):
How research is structured across the submitted unit (including research groups or sub-
units), to provide context for assessing the submission.
The submitting unit’s research objectives during the assessment period and over the next
five years, including a review of the submitting unit’s research plans described in REF 2014.
How the unit has sought to enable and/or facilitate the achievement of impact arising from
their research and how they are shaping and adapting their plans to ensure that they
continue to support the vitality and sustainability of the unit’s impact in the future. The
submitting unit should describe how the selected case studies relate to their approach to
achieving impact.
76
The submitting unit’s approach to supporting interdisciplinary research, where applicable, in
the context of the unit’s research strategy. The sub-panels will give due credit where these
arrangements have enhanced the vitality and sustainability of the research environment.
Units where this is not applicable will not be disadvantaged in the assessment.
Within the context of the institution’s strategy, how the submitting unit is progressing towards
an open research environment, including where this goes above and beyond the REF open
access policy requirements, and wider activity to encourage the effective sharing and
management of research data, as appropriate to the discipline. Consideration of
reproducibility should also be included where relevant to the discipline.
Within the context of the institution’s approach, how the unit supports a culture of research
integrity, and ensures that research is conducted according to appropriate ethical, legal and
professional frameworks, obligations and standards.
Main Panel D supplementary criteria REF5b: Unit context and structure, research and
impact strategy
347. Main Panel D is aware that the development of an open research environment poses
specific issues for the disciplines covered by its sub-panels. In this context, evidence of an open
research environment could include, but is not limited to:
any contribution from the unit to open access debates, processes and structures within the
subjects covered by the Main Panel
any contribution from the unit to the development of an open research culture within the
subjects covered by the Main Panel, especially where these reflect on the distinctive
nature of research and research data as generated by the relevant subject communities
(e.g. IP and licensing constraints/issues). This could include, for example, a range of
mechanisms to share research and research data openly, as appropriate to the discipline.
REF5b, Section 2: People
348. This section should provide evidence about: staffing strategy and staff development within the
submitted unit, including evidence of how the staffing strategy relates to the unit’s research and
impact strategy and physical infrastructure; support for early career researchers and career
development at all stages in research careers; support mechanisms for, and evidence of the training
and supervision of, postgraduate research (PGR) students; and evidence of how the submitting unit
supports and promotes equality and diversity. Particular attention will be paid to how submitting units
address all relevant aspects of support for equality and diversity (which should be taken to refer to all
protected characteristics) within their submissions. There should be synergy between the strategies
77
and structures set out in this section and the institution’s code of practice, which the panels will be
able to access on request.
Staffing strategy and staff development
349. This may include (but is not limited to):
staff development strategy for all staff pursuing a career in research at all stages of their
careers, including the use of mentoring, probation and appraisal and training, and the unit’s
implementation of the Concordat to Support the Career Development of Researchers
the unit’s staffing and recruitment policy and evidence of its effectiveness, including the
pattern of staff recruitment over the assessment period, the balance between short-term and
long-term contracts among Category A eligible staff, and how the demographic profile of the
unit affects current and future management of research activity, including succession
planning
evidence of how individuals at the beginning of their research careers are being supported
and integrated into the research culture of the submitting unit, including the contribution of
postdoctoral researchers to the unit (where appropriate)
the policy for research, impact leave/sabbatical leave for all staff at all stages of their careers
(including fixed-term and part-time staff)
evidence of procedures to stimulate and facilitate exchanges between academia and
business, industry or public or third sector bodies, for example, through the recruitment or
secondment of research staff
how the unit recognises and rewards staff for carrying out research and for achieving impact,
and how the unit specifically supports and enables staff to achieve impact from their
research.
Research students
350. Submitting units are invited to provide evidence of the quality of training and supervision of
PGR students and how the unit has developed a research culture into which research students are
fully integrated and prepared for further research activity. This may include (but is not limited to):
the approach to recruitment of doctoral research students, including those with protected
characteristics
evidence of studentships from major funding bodies (the sub-panels recognise the
challenges of recruiting doctoral students in the current funding environment)
details of monitoring and support mechanisms linked to evidence of progress and of
successful completions
78
details of the support provided to research students in terms of skills development and
preparation for their future career.
Equality and diversity
351. Submitting units are invited to provide evidence of their commitment to equality and diversity in
the recruitment and support of staff with significant responsibility for research and research students,
including the strategies, activities and collaborations that support equality and diversity and enable
staff and research students drawn from a wide cross-section of society to engage in research. All
relevant protected characteristics should be considered, and the submitting unit may wish to include
(but is not limited to) evidence of:
study leave arrangements (including supporting data where relevant)
arrangements for supporting flexible and/or remote working
the career pathways for part-time and fixed-term staff
how conference attendance or other necessary travel to support research is facilitated for
staff and research students with caring responsibilities, ill health etc.
how equality and diversity considerations are taken into account with regard to support for
submission of funding applications, access to internal funds, research-related promotion and
reward procedures, recruitment for research-related leadership roles, conference
attendance, sabbaticals and training
support for staff and research students returning from periods of leave (including parental
leave) or ill health, managing long-term illness, or with caring responsibilities
support for staff with protected characteristics (e.g. disabilities) to enable them to research
productively
the submitting unit’s approach to supporting the wellbeing of its staff and research students.
352. The submitting unit should also demonstrate how it has paid due regard to equality and
diversity issues in the construction of its REF submission, including the selection of the output
portfolio, and how data on the distribution of outputs across staff relate to the unit’s approach to
supporting equality and diversity. Units should explain how this approach relates to the processes
set out in their institution’s code of practice.
Main Panel A supplementary criteria REF5b: People
353. Where relevant, submitting units may also wish to include details of:
effective integration of clinical academics and NHS-employed active researchers
79
research career development of both non-clinical and clinical researchers
role of clinical researchers.
Main Panel C supplementary criteria REF5b: People
354. All sub-panels in Main Panel C recognise the role of professional and other doctoral
qualifications and their contribution to the vitality of the research environment. To obtain a clear
understanding of the nature of the research environment, units submitting in UOAs in Main Panel
C are asked to disaggregate the total number of doctoral degrees awarded as reported in REF4a
for each year in the assessment period into PhDs and research-based professional doctorates.
This information should be included as part of the ‘People: research students’ section of the
REF5b template. The disaggregated data should be presented in tabular format, reported in
academic years according to the standard data in section REF4a. The total disaggregated data
should sum to the totals reported in REF4a.
REF5b, Section 3: Income, infrastructure and facilities
355. This section should provide information about the income, infrastructure and facilities
pertaining to research and research impact, including but not limited to:
research funding and strategies for generating research income, including that allocated as
part of larger research consortia, links between research funding and high-quality research
output or impact, and major and prestigious grant awards made by external bodies on a
competitive basis. Allowance will be made for disciplines that find it more difficult to attract
research funding because of the nature of the research, and where more early career
researchers are involved
organisational infrastructure supporting research and impact, for example, evidence of areas
where there has been significant investment, or through the development of research
clusters that focus on distinctive areas of work, which may include the delivery of highly
impactful research
operational and scholarly infrastructure supporting research and impact within the submitting
unit, including technical and support staff, estate and facilities, advanced equipment, IT
resources or significant archives and collections
how any relevant equality and diversity issues have been addressed, for example in relation
to support for acquiring research funding, or support for accessing scholarly or operational
infrastructure
how infrastructure, facilities and expertise are utilised in relation to impact activities
80
the nature, quality, provision and operation of specialist research infrastructure and facilities
evidence of cross-HEI shared or collaborative use of research infrastructure including the
use of major research facilities both in the UK and overseas
significance of major benefits-in-kind (including, for example, donated items of equipment,
sponsorships secured, or other arrangements directly related to research).
Main Panel B supplementary criteria REF5b: Income, infrastructure and facilities
356. For Sub-panels 8 and 9 only: data should be provided on usage within the assessment
period (1 August 2013 to 31 July 2020) of major national and international facilities not supported
by the Research Councils which was awarded to an investigator in the submitted unit after
competitive review by a panel of internationally recognised experts. The information should be
provided for each facility in terms of the time awarded together with the total cost, where available.
Main Panel D supplementary criteria REF5b: Income, infrastructure and facilities
357. Submissions should, where possible, detail funding that has been received through sources
not reported in Higher Education Statistics Agency (HESA) returns, such as commissions from
artistic organisations and other sources, including from overseas, and how these relate to the
research activities, outputs and/or impact of the submitting unit.
REF5b, Section 4: Collaboration and contribution to the research base, economy and society
358. This section should provide information about collaboration and contribution, including:
the arrangements, support in place for and effectiveness of research collaborations,
networks and partnerships, including joint research projects with academic colleagues in
other institutions, locally, nationally or internationally and indicators of their success
evidence of how staff interacted with, engaged with or developed relationships with key
research users, beneficiaries or audiences in the period 2014 to 2020 to develop impact from
the research carried out in the unit and how these collaborations have enriched the research
environment
wider contributions to the economy and society, including evidence of the wider activities and
impact of research carried out in the unit that is not captured in the impact case studies
how the unit engages with diverse communities and publics through its research
81
evidence of the unit’s contribution to the sustainability of the discipline, support for and
exemplars of interdisciplinary research, and responsiveness to national and international
priorities and initiatives
indicators of wider influence, contributions to and recognition by the research base including,
but not limited to:
o journal editorship
o participation on grants committees
o fellowships
o prizes
o membership of Research Council or similar national and international committees
o invited keynotes, lectures and/or performances, or conference chair roles
o refereeing academic publications or research proposals
o co-operation and collaborative arrangements for PGR training, including whether these
have received formal recognition nationally or internationally.
Main Panel A supplementary criteria REF5b: Collaboration and contribution to the
research base, economy and society
359. Where applicable, submitting units should identify the number of staff meeting the definition
of Category C staff (‘Guidance on submissions’, paragraphs 374 to 376) on the census date, 31
July 2020, and describe their contribution to the research environment and, where relevant, the
unit’s submission.
360. Submitting units should also provide evidence of:
the approach to encouraging and developing best practice in undertaking research that is
reproducible, including any papers that are reproducing key papers in the field
the extent of collaboration or integration with external organisations such as health or
social care services (e.g. NHS and social care structures) and/or with industry or
government bodies, where appropriate.
Main Panel D supplementary criteria REF5b: Collaboration and contribution to the
research base, economy and society
82
361. Main Panel D would prefer to see Section 4 of the template constructed as a narrative
rather than a series of lists.
Environment data (REF4a/b/c)
362. ‘Guidance on submissions’ (Part 3, Section 4) sets out quantitative data relating to the
research environment to be included in all submissions (REF4a/b/c). Sub-panels will use the data in
the context of the information provided in the environment template (REF5b) to inform their
assessment. Data on research doctoral degrees awarded (REF4a) will be used to inform the sub-
panels’ assessment in relation to ‘research students’ (Section 2: People). Data on research income
(REF4b/c) will be used to inform the sub-panels’ assessment in relation to ‘Section 3: Income,
infrastructure and facilities’.
363. Data on both doctoral degrees awarded and research income will be considered in the context
of the narrative provided in the REF5b template, and taking account of the size of the submitting
unit, its areas of specialism, its research groups, research strategy and different levels of research
funding available in different fields.
364. The sub-panels do not require quantitative data provided by institutions in REF4a/b/c to be
reported by research group.
83
Part 4: Panel procedures
Panel competence to do business
365. Each main and sub-panel will consider, confirm and document its competence to do business
at the start of each assessment meeting, taking into consideration the range of expertise as well as
the numbers of panel members present.
366. Where there is a foreseen absence of a sub-panel chair at a main panel meeting, the main
panel chair will consider whether it requires the attendance of the deputy sub-panel chair in order to
be competent to do business. Attendance of the deputy sub-panel chair at main panel meetings will
only be allowed in this case, and at the discretion of the main panel chair.
Dealing with absences of the chair
367. Each main and sub-panel will elect a deputy chair for planned and unforeseen absences of the
chair, and in cases where there is a major conflict of interest for the chair. In the absence of the
chair, the deputy will chair meetings of the panel. Where both the chair and deputy declare a conflict
of interest in the same institution, the panel will nominate one of the remaining members to officiate
in that instance.
Conflicts of interest
368. All REF main and sub-panel chairs, members, assessors, observers, secretaries and advisers
will observe the arrangements for managing potential conflicts of interest set out in Annex D.
Confidentiality arrangements
369. All REF main and sub-panel chairs, members, assessors, secretaries, advisers and observers
are bound by the terms of the REF confidentiality arrangements as detailed in Annex E. These
arrangements have been put in place to enable the effective management and operation of the REF,
and for the protection of panel members.
Data protection
370. All REF main and sub-panel chairs, members, assessors, secretaries, advisers and observers
shall ensure that personal data are kept securely, maintained confidentially and used only for the
purposes set out in the Guidance on Submissions and the Panel Criteria and Working Methods. All
personal data are subject to the provisions of the Data Protection Act 2018 (DPA2018) and the
General Data Protection Regulation (GDPR), and all persons storing, handling or processing
personal data for the REF shall adhere to the principles and requirements set out in the DPA2018
and the GDPR.
84
Part 5: Panel working methods
Main panel working methods
371. Each main panel has worked with its sub-panels to review and define common assessment
criteria, as set out above. Main panels will work with their sub-panels throughout the assessment
process to ensure that the published procedures are followed and that the overall assessment
standards are applied consistently. Each main panel will also be responsible for deciding on the
quality profile to be awarded to each submission in each of the UOAs in its remit, after
recommendations have been made by the sub-panels.
372. Each main panel will work with its sub-panels as follows:
a. Main panel meetings. The main panels will meet regularly throughout the planning and
assessment phases to ensure close working and communication between sub-panels,
to identify issues for early action, seek advice on handling specific cases, resolve
emerging differences, share developing good practice and provide assurance on the
procedures being followed. Sub-panel chairs will report to the main panel meetings on
general progress and on the implementation of working methods, particularly on issues
where cross-panel consistency is significant, including:
interdisciplinary research outputs
cross-referrals
the range of output types
impact case studies
double-weighted outputs.
b. Main panel member attendance at sub-panel meetings. The chair and members of
the main panel will attend some meetings of sub-panels, to provide assurance that
practices are consistent across the group of sub-panels:
The members of the main panel advising on interdisciplinary research will in
particular be engaged in calibration processes relating to interdisciplinary research
outputs, supporting the interdisciplinary advisers on the sub-panels, and advising on
the consistency of assessment standards for interdisciplinary research.
The international members of the main panel will, in particular, be engaged in sub-
panel calibration processes and in the formation of quality profiles, to ensure
consistency with international standards.
Main panel user members will, in particular, be engaged in briefing and calibration
among sub-panel user members and assessors, providing support and focus for
85
them, and advising on consistency of method and efficient use of expertise and
knowledge in assessing impact case studies.
The main panel chair and main panel members will attend a sample of sub-panel
meetings as agreed with the main panel, especially at an early stage in the
assessment process.
c. Advice and support to panels. A group of panel advisers and panel secretaries will be
appointed to support the work of each main panel and its sub-panels. The secretariat
will be briefed and trained in providing advice and guidance to their group of panels on
the assessment procedures. Each member of the panel secretariat will work with several
sub-panels within a main panel, providing consistent support and advice across them
and providing feedback to the main panel chairs as appropriate.
d. Cross-panel appointments. Individual academic members and assessors, and
individual user members or assessors, may be appointed to work with more than one
sub-panel, particularly where there are substantial overlaps between UOAs, to
contribute to consistency in the assessment of work on the boundaries. In considering
the selection and appointment of further members and assessors, the main panel will
identify where such boundaries could benefit from joint appointments.
e. Calibration exercises. Each main panel and its sub-panels will undertake calibration
exercises at an early stage in the assessment to develop a common understanding of
the assessment standards and the application of the quality levels. International and
user members of the main panel will participate in these exercises to assist in
benchmarking judgements. The main panel chair and members of the main panel will
attend a selection of the sub-panel meetings that deal with calibration exercises, and
main panels will receive and discuss reports from sub-panel chairs on these exercises.
f. Reviewing emerging assessment outcomes. The main panels will review the
emerging assessments at UOA level from their sub-panels during the course of the
assessment phase, to support the consistent application of assessment standards. To
facilitate this review, the group of sub-panels within each main panel will adopt a
common process for the formation of each of the three sub-profiles and a common
sequence in which each sub-profile will be formed. In considering the emerging
assessment outcomes from sub-panels, the main panels will seek advice from the
international members about the application of internationally referenced standards, and
from the user members about the assessment of impact.
g. Deciding on the outcomes. When considering the quality profiles recommended by its
sub-panels, each main panel will confirm that the published assessment procedures and
criteria have been applied by the sub-panels, and that the sub-panels have consistently
applied the overall standards of assessment. The main panels recognise that there may
be a range of overall profiles across their respective UOAs reflecting the relative
86
strength of the disciplines in the UK. Each main panel will require that any substantial
differences in the overall profiles for each UOA are investigated and understood before
approving the quality profiles recommended by its sub-panels. Where the
recommendations of a given sub-panel for the overall results for that UOA are at
substantial variance from the other sub-panels, the sub-panel chair will need to justify
this to the main panel with reference to external evidence where available.
373. In addition to the main panels’ approaches to ensuring consistency within each group of sub-
panels, to support appropriate consistency across the four main panels:
a. Generic assessment criteria and working methods across all main and sub-panels have
been developed, as set out in ‘Guidance on submissions’ and throughout this document.
These include standard weightings for each of the elements of the assessment (outputs,
impact and environment), generic criteria for assessing each element, a consistent
approach to staff circumstances, and consistent working methods and procedures.
b. The four main panel chairs, the REF director and the panel advisers will meet regularly
throughout the assessment phase to discuss progress, identify issues for early action
and inform the work of the main panels. This will include planning and reporting on
calibration exercises, including cross-main panel calibration, and reviewing emerging
and final outcomes across the four main panels. Specific actions will be identified to
support consistency across those sub-panels in different main panels that have a
significant overlap (for example, sharing of some of the material used in calibration
exercises, and identifying opportunities for appointing assessors to work across those
sub-panels).
Sub-panel working methods
374. Each sub-panel will be responsible for assessing submissions in its UOA, applying the
published criteria and working methods, and recommending the outcomes to the main panel. This
section sets out how the sub-panels will undertake their work at each stage of the assessment
process.
Appointing the full assessment phase sub-panel
375. In early 2020, the sub-panels will examine institutions’ submission intentions and identify their
requirements for further members and assessors to ensure the sub-panel has the breadth and depth
of expertise required to carry out the assessment. These members and assessors will be appointed
prior to the start of the assessment to ensure that sub-panels have access to appropriate expertise
to reach robust and valid judgements with regard to the material anticipated in submissions. The
procedures for appointing members and assessors are described in paragraphs 27 to 36.
376. Sub-panels will consider the breadth of work in actual submissions early in the assessment
phase in 2021 in order to confirm that the sub-panel collectively has the breadth and depth of
87
expertise to assess the work submitted. Where necessary, sub-panels may recommend the
appointment of further additional assessors or, exceptionally, request that specific parts of
submissions should be cross-referred to another sub-panel (as described in paragraphs 399 to 404
and indicated, where appropriate, in the UOA descriptors).
377. Each sub-panel will include research user members and impact assessors, with appropriate
expertise to contribute fully to the assessment of the impact element of submissions, alongside
academic members of the sub-panels. The research user members and impact assessors will be
appropriately briefed (for example, with respect to the details of the REF process and key issues,
including equality and diversity) alongside the sub-panel members and output assessors.
Allocating work
378. The sub-panel chair, consulting with the deputy chair, interdisciplinary adviser(s) and sub-
panel members, as appropriate, will allocate work to members and assessors with appropriate
expertise, taking account of any conflicts of interest and the sub-panel’s approach to identifying
outputs with significant material in common (see paragraphs 214 to 215). This allocation may be at
the level of individual or groups of outputs, individual or groups of impact case studies, and whole
environment templates.
379. Each member and assessor on a sub-panel will be allocated a significant volume of material to
assess, so that each member and assessor makes a significant contribution to the sub-panel’s
overall recommendations.
380. Each impact case study will be allocated to at least one academic member and one user
member or assessor, wherever practicable. User assessors will be allocated impact case studies,
and may be allocated relevant parts of the environment template. User members may in addition
to impact case studies be allocated whole environment templates and/or outputs in particular
areas where they are willing and have appropriate expertise to assess them.
381. Where a sub-panel cross-refers parts of a submission to another sub-panel for advice, the
procedures in paragraphs 399 to 404 will be followed. Where a sub-panel refers outputs in a
language other than English to external specialist advisers, the procedures in ‘Guidance on
submissions’ paragraphs 285 to 287 will be followed.
Calibration of assessment standards
382. Sub-panels will undertake early calibration exercises with respect to outputs, impact and
environment, to ensure sub-panel members and assessors develop a common understanding of the
quality levels. The calibration exercises will be based on samples of a range of outputs (whether
submitted to the REF or sourced from elsewhere by panel members), on samples of submitted
impact case studies and environment templates.
383. In addition to sub-panel members, the assessors who will subsequently be involved in
assessing either outputs or impact will take part in the relevant calibration exercises.
88
384. After these initial calibration exercises, the sub-panels will continue to discuss the application
of the quality levels and will keep under review the scoring patterns of members and assessors, to
ensure consistency in the sub-panel’s standards of assessment.
Assessing submissions
385. Sub-panels will assess all of the components of submissions: research outputs, impact and
the research environment. This reflects an underpinning principle that sub-panels will assess each
submission in the round. They will not make collective judgements about the contributions of
individual researchers. Sub-panels will make collective judgements about the range of submitted
information in order to develop the sub-profiles and recommend the overall quality profile, for each
unit being assessed.
386. All the outputs listed in submissions will (unless prevented by reasons beyond a sub-panel’s
control) be examined by panel members and/or assessors. They will be examined with a level of
detail sufficient to contribute to the formation of a robust sub-profile for all the outputs in that
submission. In doing so panels will take into account additional information where relevant (as
described above), but expert review of the outputs will remain the primary means of assessing them.
387. Sub-panels will examine all the submitted case studies, and all the information submitted in
the environment template together with the standard data analysis.
388. Sub-panels will meet during the course of the assessment phase to discuss their assessment
of each element of submissions. Assessors will attend those meetings at which the relevant element
of submissions is being discussed, so that they contribute fully and on an equal basis with members,
to the development of the relevant sub-profile.
389. During the course of the assessment, the sub-panels will be asked to draw attention to any
data they would like the REF team to verify through an audit. These data will be investigated by the
REF team (in addition to the REF team auditing a proportion of submitted information from each
institution, as described in ‘Guidance on submissions’, paragraphs 91 to 97).
Assessing interdisciplinary work
390. For the purposes of the REF, interdisciplinary research is understood to achieve outcomes
(including new approaches) that could not be achieved within the framework of a single discipline.
Interdisciplinary research features significant interaction between two or more disciplines and/or
moves beyond established disciplinary foundations in applying or integrating research approaches
from other disciplines.
391. Submitting HEIs should identify those outputs that they consider meet the definition of
interdisciplinary research, as set out in ‘Guidance on submissions’ (paragraph 273) and repeated in
this document in paragraph 390, to draw this to the attention of the sub-panel. Outputs flagged as
interdisciplinary will be assessed on a fair and equal basis and will be neither advantaged nor
disadvantaged in the assessment. Sub-panels may identify outputs as interdisciplinary that have not
89
been flagged as such by the submitting HEI, to enable consideration of the most appropriate means
of assessing the output. Additionally, there will be no disadvantage in the assessment where a sub-
panel considers that a flagged output does not meet the definition of interdisciplinary research and
the output will be assessed on a fair and equal basis with other submitted outputs. In assessing
interdisciplinary outputs, sub-panels will make use of the criteria referred to in paragraph 196.
392. The interdisciplinary identifier will allow panels, working with their IDR advisers, to consider the
most appropriate means of assessing the output. It is distinct from the cross-referral process, which
is set out below in paragraphs 399 to 404. Flagging an output as interdisciplinary will not trigger
cross-referral of the output; cross-referral may be one of the assessment routes followed. The
panels do not anticipate that all interdisciplinary outputs will require cross-referral and, conversely,
expect that cross-referred outputs will not all necessarily meet the REF definition of interdisciplinary
research.
393. Given the anticipated diversity of both the interdisciplinary submissions and the sub-panels, it
is not appropriate to prescribe a single approach to assessing interdisciplinary outputs. The
processes will be developed by the individual sub-panels and will be tailored to the submissions they
receive. However, there are a number of mechanisms in place to ensure that interdisciplinary
outputs are assessed consistently across the panels, as set out in paragraphs 396 to 398.
394. Each sub-panel will have members who have experience of interdisciplinary work. Where
appropriate, this expertise will be augmented with the appointment of additional members and
assessors. Sub-panels are confident that they can assess such work, and the appointment of the full
membership for the assessment phase will seek to ensure that sub-panels have access to
appropriate expertise to reach robust and valid judgements with regard to submitted material.
395. All sub-panels will have at least two members appointed as interdisciplinary advisers. The
interdisciplinary advisers will offer guidance to the sub-panels in their assessment of interdisciplinary
outputs to enable their robust and valid assessment. This may include advising on the allocation of
outputs and the calibration and moderation of scoring. Interdisciplinary advisers will not necessarily
be expected to assess all interdisciplinary outputs submitted to their panel.
396. The interdisciplinary advisers will work in a network with their counterparts on other sub-panels
(across all four main panels) to review outcomes from the initial calibration of interdisciplinary
outputs and may be involved in the joint consideration of outputs with other advisers from the
relevant sub-panels, as required during the assessment process. The network of interdisciplinary
advisers will meet at key points during the assessment phase, which will provide a forum for
reviewing joint working arrangements and identifying wider expertise requirements. IDAP will provide
advice and support to the network on these aspects during the calibration and assessment phases.
397. The sub-panels’ approach to assessing outputs identified as interdisciplinary will be reviewed
across the main panels, to ensure an appropriate consistency of approach. Analysis of the emerging
and final scores for the group of outputs identified as interdisciplinary will be conducted by IDAP
during and upon completion of the assessment process.
90
398. IDAP will have a role in overseeing the assessment of interdisciplinary work to ensure that
agreed principles and process for assessment are applied and that there is consistency in approach
across panels. IDAP will not advise on the assessment of individual outputs, but will advise on
process, and provide advice and support for cross-panel calibration. The main panel interdisciplinary
leads will join the membership of IDAP during the assessment phase.
Cross-referral of parts of submissions
399. The sub-panels’ preferred approach is to assess work within the sub-panel to which it was
submitted and, informed by the survey of submission intentions, to appoint further members and
assessors where required to enable this. In cases where, in the sub-panel’s opinion, the sub-panel
and its appointed assessors do not have the required expertise to assess specific parts of
submissions, those parts of submissions may be cross-referred to other sub-panels for advice.
400. The submitting HEI may request that specific parts of submissions should be cross-referred to
another sub-panel for advice. The sub-panels will consider such requests and decide upon the most
appropriate means of assessing the material in question:
a. Where the sub-panel considers there is sufficient expertise within the sub-panel to reach
a robust judgement, the work will be assessed within the sub-panel. The sub-panels
expect that this will normally be the case, except where the UOA descriptors indicate
specific arrangements for cross-referral.
b. In those instances where the sub-panel does not consider it contains the appropriate
expertise, it may cross-refer the work to an appropriate sub-panel for advice (whether
within or outside the same main panel). The REF director will work with the main panels
to ensure consistency of approach in cross-referring work across the sub-panels.
401. In addition to considering requests made by institutions, sub-panels may identify specific parts
of submissions that it considers should be cross-referred to another sub-panel, and request that
such work should be cross-referred.
402. The sub-panels’ approach to cross-referral will be discussed within the main panels, to ensure
an appropriate consistency of approach.
403. Entire submissions may not be cross-referred. Specific outputs may be cross-referred. The
original sub-panel will specify the scope of advice that it is seeking. This will be limited to advice
relating to the quality of outputs. It may not include advice on other matters such as the contribution
of a co-author or double-weighting of outputs. In exceptional cases, sub-panels may cross-refer
impact case studies for advice (including advice related to the quality threshold for the underpinning
research).
404. Where parts of submissions are cross-referred, advice will be sought and given on the basis of
any specific assessment criteria and procedures (including, for example, use or not of citation data)
for the UOA in which the work was originally submitted; cross-referred parts of submissions will be
91
assessed on the same basis as work which is not cross-referred. The original sub-panel will retain
responsibility for recommending the quality profile for all work that was submitted in its UOA.
Developing and recommending quality profiles
405. Sub-panels will develop a sub-profile for each of the three elements outputs, impact and
environment of each submission.
406. Outputs sub-profile. Each output listed in a submission will be assessed against the quality
levels: 4*, 3*, 2*, 1* or ‘unclassified’. The outputs sub-profile will be formed by calculating the
percentage of outputs listed in a submission that are assigned at each quality level, with each output
contributing an equal proportion to the sub-profile. The following exceptions and rules apply:
a. Any submitted output that is found to be ineligible will be entered into the outputs sub-
profile as ’unclassified’.
b. Where a submitted member of staff is found to be ineligible, that member of staff and the
outputs attributed to them will be removed from the submission; those outputs will not
contribute to the outputs sub-profile (see ‘Guidance on submissions’, paragraphs 136 to
137).
c. Any outputs that are ‘missing’ from a submission (that is, where fewer outputs have
been submitted than the number required, as specified in paragraph 205 of Guidance
on submissions’, and where no staff circumstances apply), will be graded as
‘unclassified’.
d. Where a request to double-weight an output has been accepted by the sub-panel, the
quality level assigned to the output will be entered twice into the outputs sub-profile.
Where a request to double-weight an output is not accepted by the sub-panel the
reserve output will be assessed. If no reserve output has been submitted, the output will
contribute to the sub-profile as a single output and one instance of ‘unclassified’ will be
entered into the outputs sub-profile (‘Guidance on submissions’, paragraph 281).
e. Where the sub-panel determines that the submitted member of staff against whom a co-
authored output is listed did not make a substantial contribution to the output, the output
will be entered into the outputs sub-profile as ‘unclassified’ (paragraph 219).
f. Where a sub-panel judges that two outputs within a submission do not contain
sufficiently distinct material and should be treated as a single output, an unclassified
score will be given to the ‘missing’ output (paragraph 214).
407. Impact sub-profile. Each case study included in a submission will be assessed according to
the definitions of the starred levels in ‘Guidance on submissions’ (Annex A). Any case studies that
are ‘missing’ from a submission (that is, where fewer case studies have been submitted than the
number required, as specified in paragraphs 309 to 310 of ‘Guidance on submissions’) will be
92
graded as ‘unclassified’. The impact sub-profile will be formed by calculating the percentage of
impact case studies listed in a submission that are assigned at each quality level, with each impact
case study contributing an equal proportion to the sub-profile.
408. Environment sub-profile: Sub-panels will assess the information provided in the environment
template (REF5b), and consider the environment data within the context of that information. Sub-
panels will build up a graduated sub-profile by assessing the range of elements in each submission,
using the starred levels defined in Guidance on submissions’ (Annex A). In the environment criteria
definitions, the main panels indicate the weighting that the sub-panels will attach to each component
of the environment template.
409. The three sub-profiles will be combined into an overall quality profile, using the weightings and
method described in ‘Guidance on submissions’ (Annex B).
410. In recommending the overall quality profile for each submission to its main panel:
a. Each sub-panel will reach a collective decision, within the framework of the exercise and
in accordance with the published statement of criteria and working methods. Each sub-
panel will debate the reasoning behind the quality profiles in sufficient detail to reach
collective conclusions, and will make recommendations to the main panel on the basis
of its collective judgement. Each sub-panel will seek to achieve a consensus on all the
overall quality profiles to be recommended to its main panel. If a consensus cannot be
achieved after reasonable effort, decisions will be taken by majority vote, with the chair
holding a casting vote.
b. Each sub-panel will confirm to the main panel that each submission has been assessed
against the published criteria for that UOA (including in cases where parts of
submissions have been cross-referred to other sub-panels for advice) and according to
the published procedures.
c. Each sub-panel will confirm that each submission has been examined in sufficient detail
to form robust judgements, and that appropriate expertise has been deployed in
assessing submissions.
Recording panel decisions
411. The panel secretariat will minute details of the procedures followed by panels, and these will
be published after the conclusion of the exercise. Panels will not make or record collective
judgements about individuals’ contributions to submissions. The panel secretariat will record the
panels’ collective judgements about the sub-profiles and overall quality profiles in respect of each
submission.
93
Annex A: Examples of impacts and indicators
1. Table 1 is intended to illustrate the wide variety of areas in which impact from research across the panels may be found to have a
positive influence on the quality of life of individuals and communities locally, nationally and internationally. These are indicative only, and
in practice much of the impact will cross boundaries between them or go beyond them. Case studies are not expected to be classified in
this way by submitting units. A searchable database of impact case studies submitted to REF 2014 can be found here:
http://impact.ref.ac.uk/CaseStudies/Search1.aspx.
2. The ‘indicators’ are listed independently of the ‘types of impact’ and are not intended to link to a specific impact example listed. The
list provides illustrative examples of indicators of both reach and significance. The panels set out their approach to assessing impact
against these criteria in Part 3, Section 3, paragraphs 287 to 324.
3. Examples of impact achieved through public engagement are integrated into the different areas of impact in Table 1. More detailed
advice on achieving and evidencing impact through public engagement can be found on the website of the National Coordinating Centre for
Public Engagement: http://www.publicengagement.ac.uk/.
4. Examples are also provided of impact evaluation frameworks used outside higher education. Impact partners may also have their
own evaluation frameworks that could be drawn upon to evidence impact.
Table 1
Areas of impact
Types of impacts
Indicators of reach and significance
Impacts on the health and wellbeing
of people, and animal welfare
Impacts where the beneficiaries are
individuals and groups (both human and
animals) whose health outcomes have
been improved, whose quality of life has
been enhanced (or potential harm
mitigated) or whose rights or interests
Outcomes for patients/users or related groups
have improved.
Public health or wellbeing has improved.
Quality of life in a developing country has been
improved by new products or processes.
A new clinical or lifestyle intervention (e.g. drug,
diet, treatment or therapy) has been developed,
Measures of improved clinical outcomes,
public behaviour or health services (lives
saved, reduced infection rates).
Measures of improved wellbeing.
Evidence from clinical trials.
94
have been protected or advocated
through the application of enhanced
policy and practice for individuals or
public health activities.
trialled with patients/users, related or other groups
(e.g. community samples), and definitive (positive
or negative) outcome demonstrated.
Patient health outcomes have improved through,
for example, the availability of new drug, treatment
or therapy, diagnostic or medical technology,
changes to patient care practices, or changes to
clinical or healthcare guidelines.
A new diagnostic or clinical technology has been
adopted.
Disease prevention or markers of health have
been enhanced by research.
Misleading health claims identified by research
are not included in food packaging.
Care and educational practices have changed.
Clinical, dietary, health or social care guidelines
have changed.
Health or social care training guidelines have
changed.
Decisions by a health service or regulatory
authority (to take, or not to take action) have been
informed by research.
Measures of improved patient/user
outcomes, public health or health services.
Documented changes to clinical and/or
public health guidelines (documented
references to research evidence in
guidelines).
Evidence of enhancement of patient/user
experience.
Evidence of take-up and use of new or
improved products and processes that
improve quality of life or animal welfare in
any given context, e.g. developing countries.
Evidence of the number of animals no longer
used in research or a specific sector (e.g.
per test, drug, laboratory, or leisure
industry).
Documented changes to animal welfare
codes or guidelines.
95
Public health and quality of life has been
enhanced through, for example, enhanced public
awareness of a health risk, enhanced disease
prevention or, in developing countries, improved
water quality or access to health and social care.
The user experience has improved.
Increased patient/user involvement in shaping and
implementing policy and practice.
Public awareness of a health risk or benefit has
been raised.
The control of diseases has changed in
developing countries.
Development or adoption of new indicators of
health and wellbeing.
Development of policy and practice with regard to
medical ethics, health services or social care
provision.
Influence on CPD and training standards.
Influence or shaping of relevant legislation.
Influencing policy or practice leading to improved
take-up or use of services.
Improved provision or access to services.
96
Animal health and welfare has been enhanced by
research.
Use of animals in research has been reduced,
refined or replaced.
Impacts on creativity, culture and
society
Impacts where the beneficiaries may
include individuals, groups of
individuals, organisations or
communities whose behaviours,
creative practices, rights, duties and
other activity have been influenced.
Collaboration with museum professionals results
in enhancements to (cultural) heritage
preservation and interpretation, including museum
and gallery exhibitions.
Co-production of new cultural artefacts, including
for example, films, novels and TV programmes.
Generating new ways of thinking that influence
creative practice, its artistic quality or its audience
reach.
Inspiring, co-creating and supporting new forms of
artistic, literary, linguistic, social, economic,
religious, and other expression.
Collaboration with public arts venues, artists and
programming professionals to produce new forms
of artistic expression.
Research-led engagement with marginalised,
under-engaged and/or diverse audiences leads to
increased cultural participation.
Arts Council England offer guidance and toolkits
for evaluating impact:
https://www.artscouncil.org.uk/quality-
metrics/quality-principles
https://www.artscouncil.org.uk/quality-
metrics/quality-metrics
https://www.artscouncil.org.uk/measuring-
outcomes/generic-learning-outcomes
https://www.artscouncil.org.uk/measuring-
outcomes/generic-social-outcomes
Testimonials from creative practitioners,
curators, media professionals.
Publication and sales figures both in the UK
and overseas, audience or attendance
figures (including demographic data where
relevant), broadcasting data and other forms
97
Developing stimuli to cultural tourism and
contributing to the quality of the tourist experience.
Improvements to legal and other frameworks for
securing intellectual property rights.
Increased understanding of local traditions leads
to enhanced cultural preservation in any given
context, for example developing countries.
New forms of artistic expression resulting in
enhancement of quality of life.
of media, download figures, or database and
website hits over a sustained period.
Evaluative reviews in the media.
Citations in reviews outside academic
literature. Independent citations in the
media, including in online documents.
Tourism data, including audience figures and
visitor numbers at exhibitions, events,
performances.
Professional evaluations of exhibitions,
performances or other outputs.
Audience/visitor/participant feedback (e.g.
through surveys, interviews or focus groups).
Impact on social welfare
Impacts where the beneficiaries include
individuals, groups of individuals,
organisations or communities whose
rights, duties, behaviours, opportunities,
inclusion, quality of life and other activity
have been influenced.
Improved social welfare, equality, social inclusion;
improved access to justice and other opportunities
(including employment and education).
Engagement with research has enhanced policy
and practice for securing poverty alleviation.
Influential contributions to campaigns for social,
economic, political and/or legal change through
engagement with civil society groups.
A beginner’s guide to evaluating social return on
investment (SROI) can be found here:
http://www.socialvalueuk.org/resource/guidance-
on-starting-out-on-sroi-2/.
Documented evidence of changes to social
policy.
Measures of improved social equality,
welfare or inclusion.
98
Changes to social policy have been informed by
research.
Changes to social policy have led to improved
social welfare, equality or social inclusion.
Research has contributed to community
regeneration or development.
Improved social and educational inclusion of
marginalised groups in any given context, for
example developing countries.
More effective integration of refugees into host
communities.
Enhanced understanding of victims’ needs in
reconciliation processes in post-conflict states.
Citations in campaign literature (e.g.
leaflets).
Evidence of public debate in the media or
other fora being influenced by the research.
Documented evidence of increased social
inclusion (e.g. participation figures).
Testimonials from civil society groups and
policymakers.
Impacts on commerce and the
economy
Impacts where the beneficiaries may
include businesses, either new or
established, the NHS, private health
and social care, agriculture or other
types of organisation which undertake
activity that may create wealth.
A spin-out or new business has been created,
established its viability, or generated revenue or
profits.
Contributing to innovation and entrepreneurial
activity through the design and delivery of new
products or services.
Decisions are made not to introduce a new
process or product as a result of research.
Social enterprise initiatives have been created.
Evidence of improved cost-effectiveness.
Evidence of service change.
Sales of new products/services.
Business performance measures (e.g.
turnover/profits, trends in key technical
performance measures underlying economic
performance).
Employment figures.
99
The costs of treatment, health or social care have
changed as a result of research-led changes in
practice.
Policies have been introduced which have had an
impact on economic growth or incentivising
productivity.
Gains in productivity have been realised as a
result of research-led changes in practice.
Research helps to stimulate foreign direct
investment (FDI).
The performance of an existing business has
been improved through the introduction of new, or
the improvement of existing, products, processes
or services; the adoption of new, updated or
enhanced technical standards and/or protocols; or
the enhancement of strategy, operations or
management practices.
Contributing to economic prosperity via the
creative sector including publishing, music,
theatre, museums and galleries, film and
television, fashion, tourism, and computer games.
Performance has been improved, or new or
changed technologies or processes adopted, in
companies or other organisations through highly
skilled people having taken up specialist roles that
draw on their research, or through the provision of
Licences awarded and brought to market;
market authorisation.
Demonstrable collaborations with industry
(including knowledge transfer partnerships,
and contracts).
Commercial adoption of a new technology,
process, knowledge or concept.
Business performance measures, for
example sales, turnover, profits or
employment associated with new or
improved products, processes or services.
Jobs created or protected.
Investment funding raised from UK and/or
non-UK agencies (venture capital/Business
Angel, and so on) for start-up businesses
and new activities of existing businesses.
Priority shifts in expenditure profiles or
quantifiable reallocation of corporate, non-
profit or public budgets.
Evidence of critical impact on particular
projects, products and processes confirmed
by independent authoritative evidence,
which should be financial where possible.
100
consultancy or training that draws on their
research.
Potential future losses have been mitigated by
improved methods of risk assessment and
management in safety- or security-critical
situations.
The strategy, operations or workplace practices of
a business have changed.
Improved support for the development of ‘small
scale’ technologies.
Improvements in legal frameworks, regulatory
environment or governance of business entities.
Better access to finance opportunities.
Enhanced corporate social responsibility policies.
More effective dispute resolution.
Alternative economic models (such as fair trade)
have been developed and adopted.
Evidence of research leading to avoidance
of negative outcomes.
Quantitative data relating, for example, to
cost-effectiveness or organisational
performance.
Tourism data, including audience figures and
visitor numbers at exhibitions, events,
performances.
Evidence of closing identified skills gaps.
Impacts on public policy, law and
services
Impacts where the beneficiaries are
usually government, non-governmental
organisations (NGOs), charities and
Policy debate has been stimulated or informed by
research evidence, which may have led to
confirmation of policy, change in policy direction,
implementation or withdrawal of policy.
Documented evidence of use in policy
debate (e.g. at a parliamentary Select
Committee, material produced by NGOs).
Citation in a public discussion, consultation
document or judgement.
101
public sector organisations and society,
either as a whole or groups of
individuals in society, through the
implementation or non-implementation
of policies, systems or reforms.
Policy decisions or changes to legislation,
regulations or guidelines have been informed by
research evidence.
A policy has been implemented (including those
realised through changes to legislation) or the
delivery of a public service has changed.
In delivering a public service, a new technology or
process has been adopted or an existing
technology or process improved.
The quality, accessibility, acceptability or cost-
effectiveness of a public service has been
improved.
(Sections of) the public have benefited from public
service improvements.
Risks to the security of nation states have been
reduced.
The work of an NGO, charitable or other
organisation has been influenced by the research.
Legislative change, development of legal principle
or effect on legal practice.
Research is used by parliamentarians to develop
proposals for new legislation through Private
Members’ Bills, or to assist scrutiny of legislation
Evidence of citation in policy, regulatory,
strategy, practice or other documents.
Direct citations of research in parliamentary
publications such as Hansard, committee
reports, evidence submissions, or briefings.
Acknowledgements to researchers on
webpages, in reports or briefings.
Evidence of influence on a debate in public
policy and practice through membership of
or distinctive contributions to expert panels
and policy committees or advice to
government (at local, national or
international level).
Quantitative indicators or statistics on the
numbers of attendees or participants at a
research event, or website analytics for
online briefings.
Qualitative feedback from participants or
attendees at research events.
Data to show close working relationships
with members or staff. For example, the
number of meetings held, minutes from
these meetings, membership of working
groups, co-authoring of publications.
102
and inform amendments to other bills such as
those introduced by government.
Research recommendations are taken up by
policymakers through membership of a
government advisory committee.
Policymakers make use of research-based critical
evidence synthesis in developing policy.
Government analysts adopt innovative
methodological or approach-based advice from
researchers.
Forms of regulation, dispute resolution or access
to justice have been influenced.
Research is used to change current processes or
services, or identify new services to be provided.
Research into the languages and cultures of
minority linguistic, ethnic, religious, immigrant,
cultures and communities used by government,
NGOs, charities or private sector to understand
and respond to their needs.
Research helps to highlight issues of concern to
parliamentarians and contributes to new analysis
of existing issues.
Research helps parliamentarians and staff to
identify inquiry topics, shape the focus of inquiries,
Testimonials from members, committees or
officials, where available.
Documented evidence of influence on
guidelines, legislation, regulation, policy or
standards.
Documented evidence of changes to public
policy, legislation, regulations or guidelines.
Analysis by third-party organisations of
parliamentary proceedings or processes, for
example studies of the passage of particular
pieces of legislation.
Documented evidence of changes to
international development policies.
Evidence of use of process/technology.
Measures of improved public services,
including, where appropriate, quantitative
information; such information may relate, for
example, to the quality, accessibility or cost-
effectiveness of public services.
Measures of improved inclusion, welfare or
equality.
Satisfaction measures (e.g. with services).
103
inform questioning of witnesses, and underpin
recommendations.
Research equips parliamentarians, their staff, and
legislative staff with new analytical or technical
skills, or refreshes existing ones.
International policy development has been
influenced by research.
Allocation and/or distribution of Official
Development Assistance (ODA) has been
influenced by research.
Policy and practice of international agencies or
institutions have been influenced by research.
Research stimulates critical public debate that
leads to the non-adoption of policy.
Formal partnership agreements or research
collaboration with major institutions, NGOs
and public bodies. Consultancies to public or
other bodies that utilise research expertise.
Evidence of engagement with campaign and
pressure groups and other civil organisations
(including membership and activities of
those organisations and campaigns) as a
result of research.
Documented evidence of changes to
international development policies.
Measures of improved international equality,
food security, welfare or inclusion.
Impacts on production
Impacts where the beneficiaries are
individuals (including groups of
individuals) whose production has been
enhanced.
Production, yields or quality have been enhanced
or level of waste has been reduced.
Research helps to create routes to international
innovation and market impact.
Research leads to improvement in productivity
and resource-use efficiency.
A new product has been recommended for
use or adopted.
Development of a new plant variety or crop
protection product which has entered the
appropriate national or international
regulatory testing system.
Evidence of improved sustainability.
104
Decisions by regulatory authorities have been
influenced by research.
More efficient production, including food
production, for example where costs have been
reduced.
Animal husbandry methods have changed.
Management practices in production businesses
have changed.
Documented changes to working guidelines.
Documented evidence of improved working
practices and/or level of production.
Impacts on practitioners and delivery
of professional services, enhanced
performance or ethical practice
Impacts where beneficiaries may
include organisations or individuals,
including service users, involved in the
development and/or delivery of
professional services and ethics.
Professional standards, guidelines or training
have been influenced by research.
Professional methods, ideas or ethics have been
influenced by research.
Professionals and organisations are able to adapt
to changing cultural values as a result of research.
Contribution to continuing personal and
professional development.
Practitioners/professionals/lawyers have used
research findings in conducting their work.
Professional bodies and learned societies have
used research to define best practice, formulate
Documented change to professional
standards, performance or behaviour.
Evidence of adoption of best practice (e.g.
by educators or law enforcement personnel).
New or modified professional standards and
codes of practice.
New or modified technical standards or
protocols.
Documented changes in knowledge,
capability or behaviours of individuals
benefiting from training.
105
policy, or to lobby government or other
stakeholders.
Workforce planning has been influenced by
research.
Educational or pedagogical practices and
methods have changed in primary, secondary,
further or higher education, within or beyond the
submitting unit.
Practices have changed, or new or improved
processes or methods have been adopted, by
individuals, companies or other organisations,
through the provision of training or consultancy.
The development of expert systems has been
influenced in areas such as medicine, human
resources, accounting, and financial services.
The quality, efficiency or productivity of a
professional service has improved.
Expert and legal work or forensic methods have
been informed by research.
Law enforcement and security practices have
changed.
Cessation of practices shown by research to be
ineffective.
Evidence of debate among practitioners,
leading to developments in attitudes or
behaviours.
Literature/web information from practitioners
and advisers, including the research findings
and how they are applied in practice.
Traceable reference to inclusion of research
in national or international industry standards
or authoritative guidance.
Traceable references by practitioners to
research papers that describe their use and
the impact of the research.
106
Impacts on the environment
Impacts where the key beneficiaries are
the natural, historical and/or built
environment, together with societies,
individuals or groups of individuals who
benefit as a result.
The environment has been improved through the
introduction of new product(s), process(es) or
service(s); the improvement of existing product(s),
process(es) or services; or the enhancement of
strategy, operations or management practices.
New methods, models, monitoring or techniques
have been developed that have led to changes or
benefits.
Policy debate on climate change or the
environment has been influenced by research.
Policy debate on the environment, environmental
policy decisions or planning decisions have been
stimulated or informed by research and research
evidence.
Improved design or implementation of
environmental policy or regulation.
The management or conservation of natural
resources, including energy, water and food, has
changed in a developing country.
The management of an environmental risk or
hazard has changed.
Changes in environmental or architectural design
standards or general practice.
Sales of new products, or improvements in
existing products, that bring quantifiable
environmental benefits.
Verifiable influence on particular projects or
processes which bring environmental
benefits.
Evidence of generic environmental impact
across a sector, confirmed by independent
authoritative evidence.
Traceable reference to inclusion of research
into government policy papers, legislation
and industry guidance.
Traceable reference to the influence of
research in planning decision outcomes.
Sales of new products or improvements in
existing products that bring quantifiable
environmental benefits.
Traceable impacts on particular projects or
processes which bring environmental
benefits.
Evidence of generic environmental impact
across a sector, confirmed by independent
authoritative evidence.
107
Influence on professional practice or codes.
Changes in practices or policies affecting
biodiversity.
The operations of a business or public service
have been changed to achieve environmental
(green) objectives.
Direct intervention, based on research evidence,
has led to a reduction in carbon dioxide or other
environmentally damaging emissions.
Increased understanding of the environmental
impact of a product or process means that it is not
adopted by industry.
Documented case-specific improvements to
environment-related issues.
Traceable reference to inclusion of research
into government policy papers, legislation
and industry guidance.
Traceable reference to impact of research in
planning decision outcomes.
Impacts on understanding, learning
and participation
Impacts where the beneficiaries are
individuals, communities and
organisations whose awareness,
understanding, participation or
engagement have been enhanced as a
result of research.
Enhanced cultural understanding of issues and
phenomena; shaping or informing public attitudes
and values.
Public interest and engagement in research has
been stimulated through, for example, the
enhancement of science education in schools.
The awareness, attitudes or understanding of
(sections of) the public have been informed, and
their ability to make informed decisions on issues
improved, by engaging them with research.
Public or political debate has been shaped or
informed by research; this may include activity
Many organisations use the Generic Learning
Outcomes (GLO) to evaluate impacts on
knowledge and understanding:
https://www.artscouncil.org.uk/measuring-
outcomes/generic-learning-outcomes.
The Heritage Lottery Fund also offers guidance
on evaluating participation and learning:
https://www.hlf.org.uk/evaluation-guidance.
108
that has challenged established norms, modes of
thought or practices.
Contributing to processes of commemoration,
memorialisation and reconciliation.
Contributing to a wider public understanding of
basic standards of wellbeing and human rights
conceptions.
Contributing to widening public access to and
participation in the political process.
Professionals and organisations have adapted to
changing cultural values.
Research has challenged conventional wisdom,
stimulating debate among stakeholders.
Increased understanding of gender roles in any
given context (e.g. developing countries) has
improved equality.
Changes to education or the school curriculum
have been informed by research.
Influencing the design and delivery of curriculum
and syllabi in schools, HEIs or other educational
institutions.
Documented evidence that public
understanding has been enhanced through
active collaborative involvement in research.
Documented evidence of policy debate (e.g.
in Parliament, the media, material produced
by NGOs).
Public debate in the media.
Documented shift in public attitude (e.g. to
sexual behaviour, or social factors in health).
Documented evidence of enhanced
awareness of health risks and benefits by
consumers.
Citation in a public discussion, consultation
document or judgement.
Citation by journalists, broadcasters or social
media.
Evidence of increased public uptake of
scientific training, through public
engagement.
Information about the number and profile of
people engaged and types of audience.
109
Research results in changes to the delivery of
vocational courses and subsequently changes to
professional practice.
Reduced gap in academic attainment for students
with protected characteristics.
Evidence of secondary reach, for example
from follow-up activity or media coverage.
Evidence of sustainability through, for
example, a sustained or ongoing
engagement with a group, a significant
increase in participation in events or
programmes or use of resources.
Evidence of engagement with campaign and
pressure groups and other civil organisations
(including membership and activities of
those organisations and campaigns) as a
result of research.
Measures of increased attainment and/or
measures of improved engagement with
science in non-HE education.
Evidence of use of education materials
arising from the research.
110
Annex B: Summary of additional information about outputs
1. This annex provides a summary table of all the additional information statements relating to
outputs that are required in submissions (in form REF2). It should be read alongside, and does not
replace, the guidance provided in ‘Guidance on submissions’ and in the relevant parts of the panel
criteria statements, as indicated in ‘Summary of additional information required about outputs’ below.
It is intended for institutions’ ease of reference in identifying the requirements for additional types of
information about outputs, across the four main panels.
2. The word limits for the additional information about outputs are common across the four main
panels, as set out in ‘Word limits for additional information about outputs’ below.
Summary of additional information required about outputs
Main Panel A
Main Panel B
Main Panel C
Main Panel D
a. Outputs that include significant material published prior to 1 January 2014 (paragraphs 214
to 215)
Panel
requirements:
All main panels: Statement on how far the earlier work was revised to incorporate
new material
Reference:
Part 3, Section 2, paragraph 215
b. The researcher’s contribution to a co-authored or co-produced output (paragraphs 216 to
220)
Panel
requirements:
Affirmation of the
author’s
contribution to the
output (selected
from the
statements
provided) only
where the author
is not the lead or
corresponding
author and the
output has 15 or
more co-authors
Sub-panels 7, 8,
10, 11 and 12:
None
Sub-panel 9:
Affirmation of the
author’s
contribution to the
output only where
the author is not
the lead or
corresponding
author and the
output has 15 or
more co-authors
None
None
111
Reference:
Part 3, Section 2,
paragraphs 221 to
225
Part 3, Section 2,
paragraph 226 to
231
Part 3, Section 2,
paragraph 231 to
232
Part 3, Section 2,
paragraph 233 to
236
c. Request to double-weight an output (paragraphs 237 to 241)
Panel
requirements:
All main panels: a supporting statement to justify the request
Reference:
Part 3, Section 2, paragraphs 242 to 243
Part 3, Section 2, paragraphs 244 to 247
d. Abstract for outputs in languages other than English (paragraph 272)
Panel
requirements:
All main panels: For all outputs in languages other than English, a short abstract to
describe the nature and content of the work. (This requirement is waived for outputs
submitted in UOA 26 (Modern Languages and Linguistics) in a language included in
the sub-panel descriptor see Part 3, Section 2, paragraph 272.)
Reference:
Part 3, Section 2, paragraph 272.
e. Information about the research process and/or content
Panel
requirements:
Statement where
this is not evident
within the output
(for non-text or
practice-based
outputs)
Statement where
this is not evident
within the output
(for non-text or
practice-based
outputs)
Identification of the
original research or
new insights
reported (for
reviews)
Statement where
this is not evident
from the output
itself (for any type
of output)
For practice-based
outputs, an
explanatory
presentation in
paper format
should be included
For software and
data sets, a full
written description
with details how to
access
Statement for any
output where the
role of the
researcher, or
research process,
is not evident
within the output
Statement on the
contribution of the
attributed author to
anthologies, edited
books, special
issues of journals
and curatorial
projects
Rationale for
grouping short
items as a single
output
112
Reference:
Part 3, Section 2,
paragraph 252
Part 3, Section 2,
paragraphs 254 to
255
Part 3, Section 2,
paragraphs 259 to
262
Part 3, Section 2,
paragraphs 263 to
273
f. Factual information about the significance of the output
Panel
requirements:
None
In UOAs 11, 12,
Factual statement
wherever available
None in UOAs 7, 8,
9 and 10
None
None
Reference:
Part 3, Section 2,
paragraph 253
Part 3, Section 2,
paragraphs 256 to
257
n/a
n/a
Word limits for additional information about outputs
Type of information
Word limit
a. Outputs that include significant material published prior to 1
January 2014
Maximum 100 words for each of
ad
b. The researcher’s contribution to a co-authored or co-
produced output
c. Request to double-weight an output
d. Abstract for outputs in languages other than English
e. Information about the research process and/or content
Maximum 300 words
f. Factual information about the significance of the output
Maximum of 100 words
113
Annex C: Main Panel D output types & submission guidance
1. Sub-panels in Main Panel D receive the widest diversity of output types across the exercise. Each submitted output needs to have a
single classification selected from this list. The purpose of the classification is to assist in the management of the collection and distribution
of outputs, the allocation of outputs to reviewers and a post-submission analysis of types of outputs submitted. The sub-panel will assess
the research content of the material submitted regardless of the classification.
2. The format for the presentation of outputs (irrespective of their classification) is flexible, within the requirement to limit the format
either to an electronic submission which is submitted via the REF submission system either as a URL, DOI, or by uploading a PDF; or as
physical material which is sent to the REF team, and which may include digital/electronic material on a media storage device e.g. USB. An
individual output cannot be submitted both electronically via the REF submission system and as a physical output.
3. It is the responsibility of the submitting HEI to ensure that any digital material submitted is accessible from a range of devices.
4. Please also cross-refer to Part 3, Section 2, paragraphs 263 to 271, and Annex B, covering ‘Additional Information’.
Type
Recommend
ed additional
statements
(making
clear the
research
process/
content/contr
ibution)
Medium
Recommended content
Artefact
Up to 300
words
DOI/URL/
PDF/USB
A succinct and coherent presentation of the research, evidencing the year and mode of
dissemination.
The material submitted should provide sufficient information to allow the panel to assess
the research process, research insights, and time and manner of dissemination.
114
This may take the form of moving image, sonic, visual or other digital media or written
text, or a combination of these, as appropriate, to enable the panel to access the research
dimensions and/or the researcher’s contribution to the artefact and to assess its
significance, originality and rigour. The panel does not normally expect to receive physical
artefacts.
Authored
books
Up to 300
words*
Physical
copy of
the
book/PDF
It is anticipated that the research will normally be evident within the submitted ‘book’ and
that no additional information is required.
Novels and poetry collections should be submitted in this category.
*For books where the research is not self-evident (e.g. Artists’ books) they should be
submitted with a 300-word statement, or submitted as ‘Artefact’ or ‘Other’.
Chapter in
book
Up to 300
words*
Physical
copy of
the
book/PDF
It is anticipated that the research will normally be evident within the submitted chapter and
that no additional information is required.
Short stories and individual poems should be submitted in this category or under journal
article.
*For chapters where the research is not self-evident they should be submitted with a 300-
word statement, or submitted as ‘Artefact’ or ‘Other.
Composition
Up to 300
words
DOI/PDF/
USB or a
printed
score
where
appropriat
e (e.g.
A succinct and coherent presentation of the research, evidencing the year of
dissemination.
The material submitted should provide sufficient information to allow the panel to assess
the research process, research insights, time and manner of dissemination.
115
large-
format
scores)
This may take the form of moving image, sonic, visual or other digital media or written
text, or a combination of these, as appropriate, to enable the panel to access the research
dimensions and/or the researcher’s contribution to the composition and to assess its
significance, originality and rigour. Where possible or appropriate, scores should be
accompanied by recordings.
Conference
contribution
Up to 300
words*
DOI/PDF
It is anticipated that the research will normally be evident within the submitted
proceedings and that no additional information is required.
*Where the Conference Contribution is other than a paper (e.g. a performance or visual
presentation), this should be submitted as ‘Performance’ or ‘Other’, and be supported by
a 300-word statement.
Confidential
report
PDF
It is anticipated that the research will normally be evident within the submitted report and
that no additional information is required.
Design
Up to 300
words
DOI/PDF/
USB
A succinct and coherent presentation of the research, evidencing the year of
dissemination.
The material submitted should provide sufficient information to allow the panel to assess
the research process, research insights, and time and manner of dissemination.
This may take the form of moving image, sonic, visual or other digital media or written
text, or a combination of these, as appropriate, to enable the panel to access the research
dimensions and/or the researcher’s contribution to the design and to assess its
significance, originality and rigour.
Devices &
products
Up to 300
words
DOI/PDF/
USB
A succinct and coherent presentation of the research, evidencing the year and mode of
dissemination.
116
The material submitted should provide sufficient information to allow the panel to assess
the research process, research insights, and time and manner of dissemination.
This may take the form of moving image, sonic, visual or other digital media or written
text, or a combination of these, as appropriate, to enable the panel to access the research
dimensions and/or the researcher’s contribution to the devices or products and to assess
its significance, originality and rigour. The panel does not normally expect to receive
physical devices or products.
Digital or
visual media
Up to 300
words
DOI/PDF/
USB
A succinct and coherent presentation of the research, evidencing the year and mode of
dissemination.
The material submitted should provide sufficient information to allow the panel to assess
the research process, research insights, and time and manner of dissemination.
This may take the form of moving image, sonic, visual or other digital media or written
text, or a combination of these, as appropriate, to enable the panel to access the research
dimensions and/or the researcher’s contribution to the output and to assess its
significance, originality and rigour.
Exhibition
Up to 300
words
DOI/PDF/
USB
A succinct and coherent presentation of the research, evidencing the year of
dissemination.
The material submitted should provide sufficient information to allow the panel to assess
the research process, research insights, and time and manner of dissemination.
This may take the form of moving image, sonic, visual or other digital media or written
text, or a combination of these, as appropriate, to enable the panel to access the research
dimensions and/or the researcher’s contribution to the exhibition and to assess its
significance, originality and rigour.
117
Edited book
Up to 300
words
Physical
copy of
the
book/PDF
/DOI
Sub-panels expect to receive edited books where the researcher has made a
demonstrable contribution to the research published (in addition to any chapter published
in the same work). Where such a research contribution is part or all of the output to be
assessed, the whole work should be submitted. Submitting units may provide a statement
(of up to 300 words) to clarify the nature of the individual’s research contribution.
Special Issues of Journals should be submitted in this category.
Journal
article
DOI/PDF
It is anticipated that the research will normally be evident within the submitted journal
article and that no additional information is required.
Short stories and individual poems should be submitted in this category or under Chapter
in book.
Patent/
published
patent
PDF/phys
ical copy
It is anticipated that the research will normally be evident within the submitted patent and
that no additional information is required.
Performance
Up to 300
words
DOI/PDF/
USB/CD/
DVD
A succinct and coherent presentation of the research, evidencing the year of
dissemination.
The material submitted should provide sufficient information to allow the panel to assess
the research process, research insights, and time and manner of dissemination.
This may take the form of moving image, sonic, visual or other digital media or written
text, or a combination of these, as appropriate, to enable the panel to access the research
dimensions and/or the researcher’s contribution to the performance and to assess its
significance, originality and rigour.
118
Research
data sets
and
databases
Up to 300
words
DOI/URL/
PDF
A succinct and coherent presentation of the research, evidencing the year of
dissemination.
Research Corpora should be included in this category.
The material submitted should provide sufficient information to allow the panel to assess
the research process, research insights, and time and manner of dissemination.
Research
report
DOI/PDF/
physical
copy
It is anticipated that the research will normally be evident within the submitted report and
that no additional information is required.
Scholarly
edition
Up to 300
words*
DOI/PDF/
Object
It is anticipated that the research will normally be evident within the submitted edition and
that no additional information is required.
* Scholarly Editions where the research is not self-evident (e.g. some Editions of Music)
should be submitted with a 300-word statement.
Software
Up to 300
words
DOI/URL/
PDF
A succinct and coherent presentation of the research, evidencing the year of
dissemination.
The material submitted should provide sufficient information to allow the panel to assess
the research process, research insights, and time and manner of dissemination.
Translation
Up to 300
words*
DOI/PDF/
Object
It is anticipated that the research will normally be evident within the submitted ‘translation’
and that no additional information is required.
*Translations where the research is not self-evident, should be submitted with a 300-word
statement.
119
Working
paper
DOI/PDF
It is anticipated that the research will normally be evident within the submitted output and
that no additional information is required.
Other
Up to 300
words
DOI/PDF/
USB/phys
ical copy
Likely to include, but not limited to:
creative projects where another item classification is unsuitable
curatorial projects
a creative writing collection (a number of related works that were published in forms
other than a book length collection)
a collection of creative and/or critical work (e.g. related articles, books, choreographic
materials, essays, dramaturgical works, films, recordings etc.) on a related topic that
address different aspects of a single project and are collectively greater than the sum
of their parts
substantial dictionary or encyclopaedia entries and groups of short items including
groups of entries
buildings
design processes / programme of research.
In all such cases, the submission should provide a succinct and coherent presentation of
the research, evidencing the year of dissemination.
This may take the form of moving image, sonic, visual or other digital media or written
text, or a combination of these, as appropriate, to enable the panel to access the research
dimensions and/or the researcher’s contribution to the output and to assess its
significance, originality and rigour.
The material submitted should provide sufficient information to allow the panel to assess
the research process, research insights, and time and manner of dissemination.
120
Annex D: Managing conflicts of interest
1. The primary purpose of REF 2021 is to produce overall quality profiles for each submission
made by institutions, which will be used by the UK higher education funding bodies in determining
the main grant for research to the institutions which they fund.
2. The REF is governed by the principles of equity, equality and transparency. In order to ensure
that these principles are adhered to, we set out below arrangements for recording declarations of
interest and avoiding potential conflicts of interest.
Declarations of interest
3. All main panel chairs and members, sub-panel chairs and members, panel advisers and panel
secretaries, observers and assessors (hereafter collectively called panel members) are asked to
make a declaration of their interest through the panel members’ website. All interests which an
objective and fair minded observer would consider could improperly influence a panel member's
assessment of an HEI's submission should be declared. An improper influence is one which
prevents a panel member judging a submission open mindedly and exclusively on its own merits. It
is important to realise that influence might be subconscious, and that the appearance of impartiality
is as important as impartiality itself. All such interests which the objective and fair minded observer
in possession of all the facts would consider raised a real (i.e. more than fanciful) possibility of
improper influence are disqualifying interests in respect of the HEI concerned. For the purpose of
REF, such disqualifying interests include:
any UK higher education institution(s) at which the individual is employed
any UK higher education institution(s) at which the individual has been employed since
January 2014
any UK higher education institution(s) at which the individual has been engaged in
substantial collaboration since the start of the assessment period (1st January 2014). This
might include organisations at which the individual has visiting lecturer/fellow/professor or
similar status or has worked on a commercial contract or consultancy basis
any UK higher education institution(s) at which the individual’s partner and/or immediate
family member (parent, sibling, child, and any person in respect of whom the individual has
an equivalently close family relationship whether biologically related or not) is employed
any financial or commercial interest in a UK higher education institution(s), including spin-out
companies
any minor interest(s), including those listed in paragraph 11 below, ruled by a panel chair to
be treated as a disqualifying interest.
121
Association with what might be considered to be rival research groups or interests or a particular
approach to or school of thought within the subject area in question will not be considered to be a
declarable interest unless the panel member is unable to judge a submission open mindedly on its
merits.
Panel procedures
4. A complete list of the declared interests of panel members and others involved in the
assessment will be prepared by the REF team and made available, in confidence, to panels when
they start their work.
5. Individuals will be asked to update the REF team regularly on any additional interests, through
the panel members’ website. Complete lists of declared interests will be updated and circulated
accordingly on an ad hoc basis.
6. Panel members may not take part in the assessment of submissions from institutions in which
they have declared a disqualifying interest. Panel members must ensure their declarations of
interests are up to date in advance of any meeting at which any institution(s) in which they have a
disqualifying interest is to be discussed. Panel members must withdraw from that part of the meeting
at which the institution in which they have a disqualifying interest is to be discussed. Withdrawals
due to disqualifying interests shall be minuted. These procedures will also be set out in the published
panel criteria.
Requests for information
7. Panel members are likely to receive numerous invitations to discuss issues concerned with
REF 2021. Although the REF team seeks improved clarity and transparency during this exercise
through the dissemination of information, we do not wish panel members to compromise their
position by entering into discussions which could be perceived to give a particular individual or
institution an unfair advantage.
8. Therefore, panel members should not discuss issues concerning individual departmental or
institutional submissions that in any way break the confidentiality agreements they have entered into
in order to work on the REF. However, they may accept invitations to talk at meetings where a
number of different institutions are represented, for example those arranged by a professional body
or subject association to discuss the REF process in general terms. If any member has concerns
over a potential conflict of interests or the propriety of a proposed action, they should discuss it with
the REF director.
9. Panel members are not expected to suspend normal relations with their colleagues and peers
during the exercise. They should not feel in any way obliged, for example, to withdraw from external
examining, or participation in appointment committees. They are, however, asked to exercise
122
caution in dealings with individual departments, or with subject associations or similar bodies, where
there is an actual or clearly inferable connection with their panel membership.
Declarations of minor interests
10. Any interest that could lead a reasonable observer to doubt the impartiality of a panel
member’s assessment of work that has been allocated to them, that is not a disqualifying interest,
must be declared by that panel member as a minor interest. Minor interests should be declared on
an ad hoc basis to the chair of the relevant main or sub-panel. Declarations of minor interests shall
be minuted.
11. Minor interests could include, for example:
A panel member supervises or co-supervises one or more doctoral students from the
submitting institution, or who went on to become an academic staff member within the
submitting institution.
A panel member was supervised as a doctoral student by a staff member who is returned
within the submission.
A panel member is co-investigator or co-holder of a grant with the submitting institution.
A panel member, or their partner or immediate family member, is employed by a ‘user’
organisation that is the focus of an impact case study.
A panel member is on the editorial board of a journal series published by the submitting
department or unit, or has co-organised a conference or conference series with the
submitting department.
A panel member has acted during the assessment period as a member of an appointment or
promotions committee for the submitting institution.
Prior to their appointment to the REF panel but during the assessment period, a panel
member has acted as an external adviser to the submitting institution on their research or
REF strategy.
A panel member acts as an external examiner for research degrees for a submitting
department or unit.
12. In each case it shall be for the chair to decide what effect the existence of a minor interest
shall have on a panel member’s participation in the assessment. These decisions shall also be
minuted. Depending on the nature of the interest, the sub-panel chair may decide:
123
that the interest should be noted by the sub-panel, but that it should not affect the panel
member’s participation in assessing the submission
that the panel member should not take sole or lead responsibility for assessing the particular
aspect of the submission affected by the interest, but may otherwise be involved in
assessing the submission
that the panel member should take no part in assessing the particular aspect of the
submission affected by the interest, but may otherwise be involved in assessing the
submission
that the interest or a group of interests relating to an institution held by a panel member
shall be treated as a disqualifying interest, and the panel member should play no role in
assessing the submission.
124
Annex E: Confidentiality and information security agreement
for REF 2021 panels
1. This document sets out arrangements for the REF panels to maintain the confidentiality and
security of information they generate and have access to throughout the REF process ('Confidential
Information'). Confidential Information means all confidential or proprietary information (however
recorded or preserved) related to the Purpose that is disclosed or made available whether before or
after the date of this agreement (in any form or medium), directly or indirectly by the REF team to the
panel member.
2. All REF main and sub-panel chairs, members, assessors, secretaries, advisers and observers
for all REF assessment and advisory panels are bound by the terms set out below. For the purposes
of this document these people will all be referred to as ‘panel members’.
3. This document deals only with the relationship between the four UK higher education funding
bodies on the one hand and panel members on the other. It does not give rise to any rights or
obligations to or from HEIs participating in the REF.
4. Nothing in this agreement prevents panel members from disclosing information after it
becomes freely available in the public domain (without the breach of any obligation of
confidentiality), or that which they are required by law to disclose, or that which was already known
and not subject to confidentiality obligations before being disclosed in the context of the REF. It
would be prudent, however, to contact the REF director in advance to discuss any such disclosure.
5. Some Confidential Information may have to be disclosed by the UK higher education funding
bodies under the Freedom of Information Act 2000 or other legislation. If any requests for
information are received, these must be passed to the REF director immediately for consideration
and action, and should not be responded to by panel members.
6. If there is any doubt with regard to any issue of confidentiality, either in general terms or in
relation to a particular piece of information, panel members should seek advice from the REF
director.
Purpose
7. The Confidential Information is being provided to panel members solely to enable panel
members to carry out their agreed duties in relation to the REF, as set out in the REF appointment
invitation and associated panel role information
3
.
3
For REF advisory panels, IDAP and EDAP, the panel role will be set out in the terms of reference as
agreed by the panel at their first meeting and reviewed as appropriate points throughout the REF
exercise. For REF main and sub-panels, the panel role is set out in the REF publication ‘Roles and
recruitment of the expert panels’ (October 2017).
125
8. For the avoidance of doubt, all information panel members acquire as a result of their
appointment is confidential to the REF team and REF panel members and should not be released or
shared in any way, either during their appointment or afterwards, to third parties without the prior
permission of the REF director.
Rationale for confidentiality
9. Subject only to any other legal obligations on the UK higher education funding bodies to
disclose further information, in order to properly manage the integrity of the REF it is necessary to
ensure that public comment from REF panels and their constituent members on individual
submissions is limited to:
the ‘Panel criteria and working methods’ publication and any REF guidance documents
directly associated with formal guidance to HEIs developing submission material
the overall assessment outcomes awarded to each submission (comprising the overall
quality profile and the three sub-profiles for outputs, impact and environment)
the ‘Panel overview reports’ detailing how the assessment was carried out, and providing
observations about the assessment and the state of research within their discipline areas.
10. Subject to any overriding legal obligation, we seek to avoid any situation in which parties not
involved in the assessment process approach or place pressure on panel members to disclose
information about the panel’s discussion of particular submissions. In other words, maintenance of
confidentiality is essential if panel members are not to be inhibited from expressing their opinions
freely in panel discussions, which is essential to the effective operation of the REF as an expert
review exercise.
11. Given the nature of the information that panel members will have access to, the confidentiality
arrangements also set out measures to prevent acts by a panel member which might, in certain
circumstances, lead to a claim being made against them or the UK higher education funding bodies
for: breach of data protection legislation; breach of a common law duty of confidentiality; defamation;
infringement of intellectual property rights in research outputs; or otherwise give rise to financial or
reputational losses for which a legal claim is made or may be made.
Panel members’ obligations
General obligations
12. Acceptance of the purpose and rationale as set out in this document is a condition of
appointment as a panel member. Panel members are required to provide written confirmation of
agreement to these terms alongside acceptance of the appointment. The chief executives of the four
UK higher education funding bodies reserve the right to terminate appointments in the event of any
breach of these terms.
126
13. Panel members shall use confidential information only for the purposes of the REF.
Confidential information must be handled in accordance with reasonable instructions given by the
REF team. In particular, the REF team may require the deletion of any confidential information or all
copies of confidential information, or to take such additional reasonable steps to preserve the
security of the confidential information as the REF team may determine. Panel members must
promptly comply with any such instructions.
14. Panel members should respect the confidentiality of the information provided in any form
(electronically or otherwise) and the discussions panel members will be privy to, by:
only using such information for the purposes of the REF criteria development and REF
assessment, as applicable
taking all reasonable steps to ensure the security of the information, and handling it in
accordance with reasonable instructions given by the REF team
keeping the information for the purposes of the REF criteria development and REF
assessment, as relevant, and securely disposing of information when requested by the REF
team
promptly reporting any incident of information loss to the REF team
not disclosing any unpublished information about the REF assessors’ or panel members’
discussion of individual or other submissions
observing protocols issued by the REF team for responding to media or other requests for
further information. This includes consulting the REF team should panel members be invited
to speak about the REF
seeking advice from the REF team if panel members have any concerns during the process
or are unsure about the appropriate course of action.
15. Confidential information shall not be disclosed to any other person except panel members and
the REF team. All reasonable steps shall be taken to ensure that other people cannot have access
to the information, whether held in paper or electronic copy. In particular:
a. It is important to remember that computer systems, and specifically email, are not
necessarily secure, and panel members shall agree to exercise appropriate caution
when using them.
b. Confidential information will be made available to panel members via secure, password-
protected systems. Passwords must not be divulged to any other person.
16. The obligations set out in this document will subsist indefinitely.
127
Annex F: List of abbreviations
CPD
Continuing professional development
DOI
EDAP
Digital Object Identifier
Equality and Diversity Advisory Panel
FDI
FTE
Foreign direct investment
Full-time equivalent
GLO
HE
Generic Learning Outcomes
Higher education
HEI
Higher education institution
HESA
IDAP
IDR
NGO
Higher Education Statistics Agency
Interdisciplinary Research Advisory Panel
Interdisciplinary research
Non-governmental organisation
ODA
PGR
Official Development Assistance
Postgraduate research
RAE
Research Assessment Exercise
REF
Research Excellence Framework
UOA
Unit of assessment